Slashdot Mirror
Building a Massive Single Volume Storage Solution?
An anonymous reader asks: "I've been asked to build a massive storage solution to scale from an initial threshold of 25TB to 1PB, primarily on commodity hardware and software. Based on my past experience and research, the commercial offerings for such a solution becomes cost prohibitive, and the budget for the solution is fairly small. Some the technologies that I've been scoping out are iSCSI, AoE and plain clustered/grid computers with JBOD (just a bunch of disks). Personally I'm more inclined on a grid cluster with 1GB interface where each node will have about 1-2TB of disk space and each node is based on a 'low' power consumption architecture. Next issue to tackle is finding a file system that could span across all the nodes and yet appear as a single volume to the application servers. At this point data redundancy is not a priority, however it will have to be addressed. My research has not yielded any viable open source alternative (unless Google releases GoogleFS) and I've researched into Lustre, xFS and PVFS. There some interesting commercial products such as the File Director from NeoPath Networks and a few others; however the cost is astronomical.
I would like to know if any Slashdot readers have any experience in build out such a solution? Any help/idea(s) would be greatly appreciated!"
register a few thousand gmail accounts and write the interface that will make writing of data to gmail inboxes invisible to the app.
Have you checked out GFS from RedHat (formerly Sistina)?
Can't you hook up 4x 7TB Xserve RAIDs to a PowerMac and use that?
Check out AFS.
Howabout the PetaBox, used by the Internet Archive ?
Livejournal developed their own distributed filesystem:
http://www.danga.com/mogilefs/
It's scalable and has nice reliability features, but is all userspace and doesn't have all the features/operations of a true POSIX filesystem, so it may not suit your needs.
I know a certain recent Zombie network that was discovered which collectively had quite a few Pbs of storage... Of course I wouldn't recommend going down that road as it leads to you know ... jail.
News Reporters Make Tasty Polar Bear Treats!
archive.org made a petabox
http://www.archive.org/web/petabox.php
There is now a company that seems to make the same design:
http://www.capricorn-tech.com/products.html
I don't know what FS they use, but apprently it is redudent.
May or may not be what you search. Quite expensive but impressive featurelist.
f tware/gpfs.html
http://www-03.ibm.com/servers/eserver/clusters/so
Meme of the day: I browse "Disable Sigs: Checked". So should you.
What are you doing on a limited budget trying to build a 1PB solution? And why are you on a budget?
Just because you are starting at 25TB doesn't mean you aren't building a 1PB solution.
You also need to figure out what kind of bandwidth you need. It's very seldom that people have 1PB of data that is accessed by one person occasionally. If Some sort of USB or 1394 connection will work you are much better off than requiring infiniband.
Like many "ask Slashdot" questions this is the last place you should be looking for help...
If you know the scale of the problem, you should consult with a company like EMC to provide the support for this thing - you WILL need it.
Clustering the disks with iSCSI or ATAoE is trivial - you can do that very easily, but the filesystem to run on top of it is where you will have problems.
PVFS - has no redundancy - Lose one node lose them all.
GFS - does not scale well to those sizes or a large number of nodes - lots of hassle with the dlm.
GoogleFS - Essentially one write only - no small (50GB) files - little or no locking.
xFS - Way too easy to lose your data.
It seems that you only have one option:
Lustre - VERY Expensive - lots of hassle with meta-data servers and lock servers.
Go with a company to take care of all this hassle - you do not have the resources of Google to deal with this kind of thing yourself.
LL
Compete File System at http://www.python.org/pycon/2005/papers/46/Compete FileSystem.pdf.
MogileFS at http://www.danga.com/mogilefs/
I never thought I'd see the day when sites were boasting a petabyte of porn. .avis -- if you sat down and watched them end-to-end, you'd have 348 years of "backdoor sliders", "dribblers to short", "pop flies", and "long balls". We live in an enlightened age.
That's over 3 million hours of
Suppose each disk has a MTBF (mean time before failure) of 500,000 hours. That means that the average disk is expected to have a failure about every 57 years. Sounds good, right? Now, suppose you have 1000 disks. How long before the first one fails? Chances, are, not 57 years. If you assume that the failures are spread out evenly across time, a 1000-disk system will have a failure every 500 hours, or about every 3 weeks!
Now, of course the failures won't be spread out evenly, which makes this even trickier. A lot of your disks will be dead on arrival, or fail within the first few hundred hours. A lot will go for a long time without failure. The failure rates, in fact, will likely be fractal -- you'll have long periods without failures, or with few failures, and then a bunch of failures will occur in a short period of time, seemingly all at once.
You absolutely must plan on using some redundancy or erasure coding to store data on such a system. Some of the filesystems you mentioned do this. This allows the system to keep working under X number of failures. Redundancy/coding allows you to plan on scheduled maintanence, where you simply go in and swap out drives that have gone bad after the fact, rather than running around like a chicken with its head cut off every time a drive goes belly up.
...what your management was thinking. I mean, I can't imagine a storage requirement that large that you can build in a distributed model that would beat on price per GB an EMC or Hitachi or IBM or whomever SAN solution. The administration and DR costs alone for something like this would be astronomical. There just isn't really a way to do something this big on the cheap. I mean, this is what SANs were developed for in the first place. Its cheaper per GB than distributed local storage ever could be.
What?
Unfortunately, I should think needing a solution which can scale up to a Petabyte (!) of disk-space and a "fairly small" budget are at odds with one another.
Maybe you need to make a stronger case to someone that if such a mammoth storage system is required, it needs to be a higher priority item with better funding?
Heck, the loss of such large volumes of data would be devastating (I assume it's not your pr0n collection) to any organization. Buliding it on the cheap and having no backup (*)/redundancy systems would be just waiting to lose the whole thing.
(*) I truly have no idea how one backs up a petabyte
Lost at C:>. Found at C.
A while ago
the reason you can't find a cheap way to do this is because it just isn't cheap.
I would look at some lessons learned from Google. If you decide to go with some sort of homebrew solution based on a bunch of standard consumer disks you will run into other problems besides money. The more disks you have running, the more failures you will encounter. So any system you setup has to be able to have drives fail all day, and not require human intervention to stay up and running(unless you can get humans for cheap too).
He asked for low cost commodity hardware. The fact that no price is mentioned and you need to contact a sales droid for a quote is an instant red-flag. I hate vendors who do not put price lists, even 'retail' prices on their product pages. I realize they may have different price levels based on quantity, but there is a value to seeing that a product is in the '$1000-$1500' range versus the '$120000-$150000' range. Having the contact sales droids who will put your name/phone number on a sales list and harrass you just to find out the price range turns me off of a lot of these outfits. I do a lot of product research and selection using the Internet. I favor outfits who allow me to get all the info online without contacting a sales rep. Many times if I cannot get the info on the web and I cannot get a price on the first phone call without providing sales lead information, I skip them.
Look. Everyone wants a Lamborgini for the price of a Chevy. Cute. Yawn. Half of the Ask Slashdot questions are people who didn't find what they want at Walmart. Despite the amazing Slashdot advice, Ask Slashdot answers have somehow failed to put EMC, IBM, HP, etc. out of business. There is no free lunch.
Just call EMC, get a rep out, and give the paperwork to your boss. Do it today instead of 5 months from now and you will have a much better holiday season.
Note to moderators and other finger pointers: I did not say to BUY from EMC, I just said to show his boss how and why to do things the right way. It does not hurt to get quotes from the big vendors, mainly because the quote also comes with good, solid info that you can share with the PHBs. Despite what you think about "evil" tech sales persons and sales engineers, you actually can learn from them.
"If you want to improve, be content to be thought foolish and stupid." - Epictetus
You may want to take a look at IBRIX systems. They do a pretty robust parallel file system that has redundancy and failover.
That violates their terms of use pretty severely. I don't know what they would do (Google's not the "suing-for-the-hell-of-it" type), but that wouldn't last very long when they found out. And they would find out. +5 Interesting? Well, curiosity killed the cat.
Rank my idea: http://www.sinceslicedbread.com/node/531
Exactly. This seems like somebody is trying to figure out a way to do something in-house which really ought to be left to either an outside contractor, or at least set up as a turnkey solution by a consultant. Given that he knows little enough about it that he's asking for help on Slashdot, I think this is yet another problem best solved using the telephone and a fat checkbook, and enough negotiating skills to convince management to pony up the cash up front instead of piddling it out over time on an in-house solution that's going to be a hole into which money and time are poured.
/. crowd for help, calling in professionals to take over for you isn't probably a bad idea.
I know people get tired of hearing "call IBM" as a solution to these questions, but in general if you have some massive IT infrastructure development task and are so lost on it that you're asking the
It's not even a question if whether you could do it in-house or not; given enough resources you probably could. It comes down to why you want to do something like this yourselves instead of finding people who do it all the time, week after week, for a living, telling them what you want, getting a price quote, and getting it done. Sure seems like a better way to go to me.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
I wrote a web application and a client in C# that uses gmail accounts as a sort of file system. using a set of email accounts as "index" accounts that use the gmail search functionality to find what you are looking for then pulling the attachment on the index to grab the parts of the file that where spread accross multiple gmail accounts in 500K chunks. it works really well. I did it for fun to see if I could. uses smtp to post the file chunks to a given set of accounts and users can donate accounts to the hive at will, increasing the overall storage size. all hosted maintained and index by gmal or any other free mail service as one big file system.
Hard disk space is doubling every 6 months - wait 5 years and you'll be able to buy a 25TB disk for $125.00.
A single raid50 of them will then give you your petabyte of storage, for around $6,000.
Then I suggest using *nothing*. It's free, and will work with the appropriate hardware and software add-ons.
A PETABYTE without redundancy? I can't imagine having that much data I didn't care about.
"A coward dies a thousand deaths, the brave but one."
Something else I forgot about is the actual hardware... you may want to take a look at the nStor products. Their hardware RAID systems are relatively economical, and you can go to fibrechannel drives with fibre connected boxes quite easily with their equipment.
As a VERY satisfied customer, I say, just buy the damned thing from EMC. There's few enough warm fuzzy feelings that SysAdmins have in this day and age, like your CE calling at 7:00am saying: "Hey, you had a few hard SCSI errors on Disk 3 Enclosure 0 Tray 0 last night, that's your production LUNs isn't it? There should be a courier there with a disk by 10, and I'll stop by to make sure things are hotsparing back properly after you replace the disk okay?" And *THIS* is just because my CE knows I can handle replacing a disk. Normally he'd come out and do that, and sit around while it re-built the Raid Group.
Yeah, EMC costs. THIS is why. The support, when needed, is top top top notch. Which would you rather have in a DR situation?
What if it is just turtles all the way down?
If designing for speed, NOT cost:
given 2PB = 1 Human Brain, non interlaced
1024 TB == 1 PB
1 TB == 1 PC Computer with 1200GB H/D, 2Gig RAM, Networking
If designing for cost, NOT speed:
1 DVD = 4.5GB
1 PB = 1024 TB = 1,048,576 GB
1 PC Computer, with a DVD like the one mementioned above.
1 Robotic CNC Arm, with DVD Gripper(tm)
1 Very Huge Wire Cage to hold DVD's like a Juke Box.
(This has been done before, but with Tapes)
Read the post that you're replying to more carefully next time.
There seems to be lots of SATA-RAID based iSCSI SAN devices available nowadays.. Some links to products I have seen:
t m
t m
h tm
:)
d uct_detail/dataframe_420.html t _53700.html?view=1&curNodeId=0
/ section/Product~Categories/category/iSCSI/options/ IPBank/drivetype/L~Series/formfactor/Integrated/in face/SATA~-~Serial~ATA
o rmat.jhtml?articleID=170702726
http://www.equallogic.com./ They make nice SATA-raid based iSCSI SAN devices with all the features you could expect (volumes, snapshots, array/volume-expansion, hotswap, redundant controllers, redundant fans, etc).
http://www.equallogic.com/pages/products_PS100E.h
14 250G sata disks, 3U, 3.5 TB of raw storage.
http://www.equallogic.com/pages/products_PS300E.h
14 500G sata disks, 3U, 7 TB of raw storage.
http://www.equallogic.com/pages/products_PS2400E.
56+ TB
Looks good. I have not yet used them myself
Another iSCSI SATA SAN possibility:
http://www.mpccorp.com/smallbiz/store/servers/pro
16 sata disks, review:
http://www.infoworld.com/MPC_DataFrame_420/produc
This company also has SATA iSCSI SAN devices:
http://www.dynamicnetworkfactory.com/products.asp
iSCSI SAN comparison:
http://www.networkcomputing.com/story/singlePageF
There are also software iSCSI target solutions for use with your own/custom hardware.
http://iscsitarget.sourceforge.net/ for building linux-based iSCSI target/SAN.
If you are familiar with iSCSI targets / iSCSI SAN devices please post your comments!
but I'm just a linux hobbyist and programmer, so take any advice I give with a grain of salt, but here's what I did for my setup at home. To start, you're looking a little over $1000 per TB. And, that's about as cheap as it gets with redudundancy. I have 8 drives in one machine, it's in a RAID 5 config, and I have a hot spare. However, if I were doing this for a mission critical application, I would have it in a RAID 6 configuration with a hot spare, and buy a hot swap cage, which would further add to the costs. Then, I would simply export the RAID 5 volume using ISCSI, and then see if there is a way to RAID all of the ISCSI volumes using a master server. I imagine that if you do it right, you could scale up such a system to a fairly large number of machines. You would probably want something faster than gigabit eithernet, probably 10,000 MB/s connecting everything together, otherwise, things could get a bit congested at the head node.
Where all this could get terribly expensive is in power requirements, it requires less power to run a cage of hard drives than it does to run a network of PC's. I'd imagine that any money you save on hardware, you would spend on your power bill. Either way, your looking at, bare minimum, about $30K to start for 25TB's, and I would add another 10K padding just to be safe, to pay for stuff like UPS (which you want), a high end switch (which you'll also need), cabling, etc. In other words, it's not cheap, and like my parent just said, it will probably be cheaper in the long run to have someone like IBM do it for you. Do you really want to be responsible for 25-1000 TB's of data?
Storage nodes: 7 x 2.8TB 2U RAID5+1 boxen with Serial ATA. The 2.8TB is logical, not physical. The OS for each of those machines is RAMDISK based (something we concocted based on what I read about the DNALounge awhile back) so it helps curb disk failures of the storage nodes themselves. We avoid disk failure by using RAID5. Of course that doesn't protect against mutiple simultaneous disk failure, but read on for more. Each of the storage nodes is exported via NBD.
Then we have a head unit, a 64-bit machine. This machine does a software RAID5 across the storage nodes using an NBD client. Essentially each storage node is a "disk" and the head unit binds and manages the sofware raid5. So let's say a whole storage node goes down (for whatever reason it does), all the data is still intact. RAID5 rebuild time over the gigabit network is about 18hrs, which is acceptable. We even have another storage box as a hot-spare.
On top of that, we have the whole cluster mirrored to another identical cluster via DRBD in a different geographic location. This is linked by Gigabit WAN. So if we have a massive disaster and lose the entire primary cluster, then we have a 2ndary cluster ready to go. We needed to purchase the Enterprise version of DRBD ($2k US) but that's worth it because they're neato guys.
We use XFS as the filesystem. This system gives us 14TB of redundant "RAID-55 with a Mirror" space. Both clusters together? $85k.
When the cluster starts running out of space (about 70% or so), we add ANOTHER cluster of similar stats to the initial one and use LVM to join the two units together.
This has scaled us to 30TB and we're pretty happy with it. The read speed is very good (hdparm says Timing buffered disk reads: 200 MB in 3.01 seconds = 66.49 MB/sec) and the write speed is about 32 MB/sec. For what our application is doing, that's a fine speed.
The filesystems going to be the hardest component of this. I know of no open-source fs that could handle this. I'm assuming this is all online storage, and there is no desire to nearline it to tape. Ideally, you'd want something that could contcatenate multiple LUNs (of RAIDed storage) without having to run through a volume manager. Nothing agaist volume managers, but it'd be another component to support. Looking at proprietary FSs, you've got CXFS from SGI, which could easily handle the PB requirement and plays nice on Linux. Sun's got QFS, which would max out at 1PB and could do the volume management bit easily. Linux support was a little flakey last time I used it, but it's a free download and evaluation, you could go get it right now.
IBM's SAN-FS would also meet the capacity needs and would have the advantage of providing nearline capability, if you're into that. Sun's SAM-FS is basically the QFS product with nearline-to-tape capability. Linux is only supported as a client OS there. Of course, if you buy the mantra that Solaris is 'open-source,' then that might not be an issue.
As for hardware with any of the above solutions, you're going to be looking at using multiple RAIDing disk enclosures of some kind. At a budget, probably SATA disks talking to the controller, and iSCSI to the host. FibreChannel to the host would be a little more costly, but might be worth it since iSCSI is just getting mature enough to be usable in production.
Dear Slashdot,
I have been tasked with (insert very difficult, very important job). This is very important to my company. I have (insert number much lower than it should be) dollars to do this. I do not want to use (insert company name specializing in this exact thing) because management thinks they are too expensive. I think I can do this (insert better/faster/cheaper/...) than said company, even though they have vastly more experience and have invested much more time and research than I have. My continued and future employment probably rests on this project. Please advise.
Maybe not
Capricorn Tech? They power the Internet Archive. "Capricorn Technologies was founded in 2004 and provides petabyte-class storage solutions for organizations worldwide. Capricorn's PetaBox technology grew out of a search for high density, low cost, low power storage systems for the world's largest data collections. Capricorn Technologies is proud to be a leader in the next data storage revolution."
...because what good is power unless you can abuse it?
Nexsan has a box called ATA Beast
Raid, Fibre Channel, 42 ATA drives per 7 RU chasis. Throw in 500GB drives and 1 parity drive for every 6 data drives and you have ~30 TB per chasis.
Fly me to the moon Let me sing among those stars Let me see what spring is like On jupiter and mars
I appreciate this might not seem like helpful advice, but...
If you've been asked to do something this by a company that can afford to buy one commercial off-the-shelf high volume storage solutions, then I honestly can't imagine any solution they try and knock up will actually work (as I'm not aware of any free software solution that's currently up to the task).
If your company doesn't have / can't raise the capital to buy a commercial system for a project of this scale, I can't possibly see how they could afford to screw up on this and go with an untested idea that could very well end up being a huge money sink they wouldn't be able to dig themselves out of - one that could doom the entire company and all it's investors given the cost it could run to.
And of course, for such a big project, they should hire people who would already know how to do something like this (which is not a dig, it's just crazy to skimp on staff when you have an ambitious project which requires large amounts of capital investment).
That said...
I were going to do large scale storage on the cheap, depending on the design of the software and the specific requirements (particularly if I was also developing the software we were going to use, or was able to set feature requirements and/or was able to make the modifications myself) I would build the largest standard file shares I could with SATA disks (using commodity hardware, hot swappable, running linux, with front loading drive bays).
The specifics of handling the load balancing (via multiple front ends, multiple mount points, pre-deteremined hashing to balance things out, proxies/caches, hooks in the file system calls, hooks in the application to talk to a controller, etc) depend entirely on the sort of application however.
It's definately likely to be far easier (and more cost effective) to have the software take care of knowing where the data is stored, rather than trying to build a single really large file share. I know at least one very known large company who've went down this route (with essentially elaborately hacked up versions of common OS software).
The downside is you have to support whatever hack you come up with to do this, but that shouldn't be an enormous amount of work (and you can probably afford to hire someone to support it full time for significantly less than the cost of a support contract for a commercial solution).
Same with disk drives — most failures will be clustered around the 57-year mark. Not that your attitude towards redunancy is wrong. Just as people sometimes die in infancy, some disk drives break down quickly. So there's a chance that you'll lose some drives from your thousand-disk system in the first year.
How big a chance? To answer that question, you need more statistics about drive failure — and a much better grasp of probability theory.
What's making your question hard is the "make it like one volume" restriction. The problem is trivial otherwise. If I were you, I'd be asking whoever tasked you with this to *really* justify on a technical level why they need it to appear as a single volume, since that makes all the possible solutions slower, more costly, and more difficult to maintain.
Chances are extremely high that what they really want is a "/bigfatfs" directory visible everywhere in which they will store many discrete items in subdirectories by project or by dataset or by user. You should convince them to let you build it from commodity machines serving a few TB each mounted as seperate filesystems underneath that umbrella directory. Then your only challenge is coherent management of the namespace of mountpoints for consistency across the environment (which there are longstanding tools for, like autofs + (ldap, nis, nis+, whatever)), and administration/assignment of new space requests within your cluster (that could be scripted to automatically allocate from the least-used volume which can satisfy the request (where least used could mean space or could mean activity hotness based on the metrics you're logging)).
11*43+456^2
The folks at the Internet Archive have already done the hard work of figuring out how to create a petabyte storage system using commodity hardware. The system works so well they started a company to sell PetaBoxes to others. Why reinvent the wheel?
What once required talent and brilliance today only requires reading a how-to file, configuring,
and rebooting.
EMC is obsolete. Their customers just haven't discovered it yet.
-I like my women like I like my tea: green-
http://www.lefthandnetworks.com/ supports all that of what the person is talking about in the article. As you add more of these units, the volumes are spread over the units you add. This means that you can add storage as you go and still have redundancy. You can configure each individual unit to use RAID 0, 1, or 5, and still get to have a volume, or many, across multiple storage units that in turn have parts of a whole voule or set of volumes. Its like haveing double mirroring, once within each individual storage unit level (which has many IDE drives in RAID 1, or 5) and then twice at the storage unit level. Of course this assumes that you have at least two storage units. And, yes, this means that to have redundancy you ahve to add them in pairs (I think) and have some storage units in one physical location and the pairs of each of those in another location for disaster recovery (fire, earthquackes, you know things can happen.)
I have worked with this units and they kick ass. You can do snapshots of entire servers quickly, given that you have the right infrastructure, set thresholds for voulmes that can be increased or reduced on the fly, brick level restoration of files!!!, etc. And of course, my respect goes to their engineers. I saw them working on one unit cause we had a really bad power failure that killed one HD. Man those guys know their stuff up and down, and I've never seen anybody type commands so complex and so freaking long at that speed! They fixed the damn thing and got 99.99999% back from limbo!
I guess their storage boxes follow the model of LVM which is pretty cool and the storage boxes run Linux!!!
Don't take my word for it, go to their website and take a look 'cause I tend to confuse people with my posts rather than pass info efficiently.
Have a good one.
===== "Every head is a different world so don't invade mine you FREAK!" smartSAGA said
MatrixStore from Object Matrix http://www.object-matrix.com/ uses commodity hardware and clusters it together to create a highly expandable, reliable and secure storage environment.
How about this:
- Use LVM on every node to make the 2TB seem like a single disk ( Assume 4 x 500GB disks )
- Use iSCSI/AoE to make the LVM volumes available on the network
- Use LVM again to merge exported volumes
- For redundancy use software raid 5 on the lvm volumes
I suspect there will be a lot of problems with efficiency but I think you should be relatively safe from hardware failures as the software raid will detect and repair them.
Anyone have any idea whether what i mention is possible/recomended?
I want to say it is 16 Tbyte offhand, but I'm not sure on that.
Short research indicates this was a limitation in 10.3, but I haven't found anything confirming or denying that 10.4 still has it.
Not that we've been looking into large amounts of Xsan storage here, but our requirements are a bit different. You can't hook >600 nodes up to the storage via fibre. Our problem is scaling out the NFS servers to be able to push all this data around.
Slashdot Patriotism: We Support our Dupes!
Stop what you are doing right now. If your architecture requires you to have one huge volume then you have architected things wrong. Imagine trying to fsck this damned thing! What about file system corruption- What the hell are you going to do when you lose a Petabyte of data because of some file system corruption? Small, sensible, easily managed smaller partitions are the way to go. Use a database to organize where given files are stored. Do something that makes sense. I have a client now who just lost a bunch of data because they used a system like this.
Having said all this- If you are still intent on finding a good file system then use AFS. It's probably your best free solution. If you want to sleep at night call EMC.
-sirket
GPFS
Take it from someone who's messed with nearly every storage product on the market, if you want something that works fairly simply, performs at approaching spindle speed ( meaning the file system is not the bottleneck - if you have 10 GB/sec. storage bandwidth, expect to see near that with proper tuning ), is very stable ( compared to most storage solutions on the market - bear in mind that most storage products are aimed at large-block sequential I/O, and fall down - either performance-wise or stability-wise - when you throw other I/O patterns or combinations of patterns at them ), and is portable across nearly any Linux distribution ( with varying amounts of difficulty, I have had to hack their kernel patches before when using a unsupported kernel ), GPFS is the one. Of course, the problem there is I believe it's pretty expensive to run on non-IBM hardware. But if you have IBM hardware ( even if it's not the hardware you're running the FS on ) or some sort of in with IBM, they'll let you have it for a song and a dance.
Having said that, Lustre is getting there. I'd say it's the equal of GPFS ( as a parallel filesystem - I believe it is even more flexible as a distributed filesystem ) in performance, probably scales roughly the same ( haven't played with it in a large installation, so can't tell you beyond looking at the architecture ), and is going to the be the biggest player on the market in the future. It's also free ( IIRC Cluster File Systems sells support, but the code is freely available ) and not tied to IBM and whatnot, like GPFS is. Of course, HP has a big connection with Lustre, but not ownership thereof.
Those are really the only two that I would consider for a serious high-performance storage project. If you don't need great performance, that's when you can start looking at things like GFS, ADIC's StorNext, Ibrix, etc.
Oh, Gautham Sastri ( of former Maximum Throughput fame ) has a newer company called Terrascale, I recall them putting on a presentation at the 2003 or 2004 ( can't remember ) Supercomputing conference ( SC2005 is coming up in a few weeks, yeah!!! ) which showed pretty good performance ( relative to the small system they were using ), not sure how they're coming along...
Anyways, good luck...and don't forget to use Iozone to benchmark the damn thing!
PC moderators can suck my White pierced, tattooed dick. If you think pride == hate, s/dick/Aryan meat mallet/g.
Why not store the data randomly in a dilithium matrix with asynchronous data transfer and AJAX? Maybe some RUBY on RAILS too - I hear that's hot right now. Of course, you'd have to make use of a couple of Heisenberg compensators configured in parallel to keep track account for any memory addressing issues, but no need to state the obvious there.
"Nokia is not a country, it's the capital of Finland!" -Moderated "Informative". Yeesh.
We at Vap-o-tech 2003 Inc. (not associated with Vap-o-tech 2001 Inc. which has closed its doors due to allegations of investor fraud) have developed ToastFS 2003. Using patented CRUMB technology and high capacity BUTTER read/write caching, we are able to turn your average loaf of Wunderbread into a 200gb storage media. Simply buy a loaf of our own specially tested Wunderbread ($250 USD) along with a USB-to-Popup Toaster interface (don't worry, USB 2.0 is more than capable of handling 120amp wall sockets without a problem, except in California). Then take our Vap-o-bake ToastFS drive and pop two pieces in. For doubled capacity, buy our Vap-o-bake ToastFSx2 drive, which takes four pieces. From a command prompt, simply type FORMAT C: and answer yes. Your new ToastFS drive will be formatted in minutes. Please note that we have 24 hour technical support via 1-900-842-8524 ext 241. Please don't hang up. Our operators in the Dutch Antilles are very busy and could take upwards of an hour to get to you.
The world's burning. Moped Jesus spotted on I50. Details at 11.
You could get a bunch of Broadcom 8 port SATA controllers, which equals about 4TB per controller. 4 or 5 controllers = 16-20TB per box, then you can run the cables into an outside drive bay enclosure and one box can control 40 500GB hard drives.
If you're not doing any processing on this, a good CPU should be able to handle the load.
If you need web hosting, you could do worse than here
I manage a small (29 dual-xeon nodes) linux cluster in a lab for my local college. A while ago I had the same problem when we ran out of storage space on the main file server.
/dev/md# blcok device;
/dev/md0 device;
/dev/md0 array, and I was done.
My solution was to use the nodes' hard disks (each one has a 120GB Ultra320 10000rpm disk) combined in a network RAID1+0 solution (we use gigabit ethernet) to get more space. With that aproach you can get as much redudancy as you need.
Heres what I did:
1. After install the network block device server (nbd-server)in each one of the nodes, I created a 100GB partition on the HD and exported then directly using the raw mode;
2. On the master node (using the nbd-client) I created a block device for each one of the nodes partitions;
3. After that I installed the linux software raid tools (mdadm) and created a small RAID1 array for each pair of nodes. I ended up with 14 100GB network RAID1 arrays each one with its very own
4. I created a big 1.4TB (14 * 100GB) RAID0 array with the 14 RAID1 ones and attached it to the
5. The final step was to create a large RaiseFS filesystem on the
You have to pay special attention to the array shutdown and startup procedures. I wrote my own scripts to take care of that for me.
Our array may seens small compared to what you are looking for, but I am pretty sure that it will scale well for arrays much larger then ours.
Good luck.
There's a reason why Terabyte storage arrays for commercial applications cost a lot of money, and why consulting services from IBM, EMC, Hitachi, etc. have the huge per-hour cost. If you/your management can't see that, you really have no business being there. Sure, anyone can throw a JBOD RAID together for a thousand bucks, but I wouldn't trust anything more important than MP3s to it.
ACHTUNG! Das computermachine ist nicht fuer gefingerpoken und mittengrabben. Ist nicht fuer gewerken bei das dumpkopfen.
In short, your stated objective smells. Not enough data.
WHAT is going to be done (database, file storage?)
HOW will it be accessed? (One large file, many smaller files)
WHEN will it be accessed? (During business hours, distributed over the day?)
AVERAGE TRANSFERS - will the whole schmear come over, selected parts?
SECURITY a concern? (Sensitive data, protected network)
BACKUP - a petabyte of tape storage is expensive, and takes quite a while to do.
POWER - do you have enough?
COOLING - ditto
SPACE - ditto - my $DAYJOB computer room is about 3000 sq ft... and we're going to be using all of it within 12 months.
That said, if you go with big drives over a lot of systems, use lots-o-nics to keep the nic from being the bottleneck. A single gig connection sounds fine, but wait until you have 100's of people going for files at once. It'll get swamped. And swear off V-SAN from Cisco. Not worth it at all.
Necessity is the plea for every infringement of human freedom. It is the argument of tyrants; it is the creed of slaves.
It sounds like a seriously ambitious project to approach...
/. archive, and laughing about how low tiny our disc storage space was back in 2005]
I second that.
Starting at 25TB to scale 1PB? And you want it cheap? If it was cheap to do that sort of thing, we'd all be lining up to get one of our own(*).
Seriously, though, you don't really specify how cheap you are expecting to get it for. What are your expectations, and just how far over-budget are the options you've looked at already? Do you really need 25TB/1PB in one volume, or could it be achieved by splitting it into smaller chunks and working out some sort of load-sharing system?
And in any case, what on Earth kind of data do they anticipate will take a petabyte of contiguous storage????
[(*) Yes, I'm aware that in X years, someone's going to be looking back at this in the
(Spudley Strikes Again!)
Just a comment about MTBF. It's often not understood, and it is one of my little pet peaves with tech producers because they don't try to correct it. MTBF is a rating for reliability to achieve lasting the warrenty period.
You have a drive that is rated 500,000 hours MTBF. Suppose you bought a drive and let it run at rated duty. Driver are normally rated to run 100% of the time, but many other devices will have duty period. Further, you run the drive until its warrenty is up. You then throw this perfectly working drive out the window and replace it. If you keep the up this pattern, then approximately once per 500,000 hours on average you should have a drive fail before the warrenty period is up. This is why it is important to not only look at the MTBF but also its warrenty period.
As a side note: In theory, you should be throwing drives out on a periodic basic. One way around this is to not buy all the same drive type and manufacturer. By having a pool of drive types, you distribute, thus minimize, risk of drive failures. Additionally, you may want to have a standard period of time for drive replacement so as to shedule your down time, as opposed to it all being unexpected.
Bel, the mostly sane.. "Of course I can't see anything! I'm standing on the shoulders of idiots." -- Me
And you don't have an answer to the question.
If you don't want to participate, don't. Stop stuffing the threads with posts about how lame everyone's questions, knowledge and motivations are.
I'm actually interested in what people have thought about this very topic, AND I'm not a petabyte database expert. So it's news to me. And probably is to you as well.
The PetaBox, as previously discussed on Slashdot sounds like just what you want...
what on Earth kind of data do they anticipate will take a petabyte of contiguous storage?
I know. They don't know I know, but I do. It's data gathered by the black helicopters, by Echelon, by Carnivore, by our very own printers, by RFID, about every movement of every single one of us... *They* do it. They.
Stupidity is an equal opportunity striker.
Fellow slashdotter Bill Dog
Wow! This is a real conspiracy theory. All I could think of was Porn. :)
exit();
File systems asside, your talking about a whole lot of hardware here! Is it really necessary to have all this data online at the same time, is it possible to store it in some other way (ie tapes) because it would probably be a whole lot cheaper!
Well, lets see... using 300Gb SCSI disks (assuming you can find raid 0 hardware to support enough disks) you can build out a 1Pb storage system with about 3,334 disks. That would set you back about $3.3Million, assuming you paid retail prices for the disks ~$1,000 / disk @ CDW today. Of course, if you orderded 3,000+ disks, I'm sure they would cut you a deal on the price.
Any hope of daisy chaining together a few dozen direct attached storage devices to a NAS server? Something like a Dell PowerVault 220 with 14 300Gb SCSI drives will set you back about $21K and give you 3.4Tb / 3U of space (RAID-5) so you would have some saftey net built in (albeit not much). Slap 10 of these on a Powervault 6000 series and you should have a ball park of 34Tb (while shy of what your looking for gets you in the right direction). Total cost around $250K - do it four times and spread the work out over four logical volumes and you should get in the neighborhood of 1Petabyte. You could then set up a redundant server structure and for $2Million you have a redundant mirrored architecture ready for one to fail and be brought up online quickly.
AF-Design, web development.
Google FS is not a real file system. It's just a bunch of API's that the program calls. The GoogleFS is not integrated into the Linux VFS. So you can't mount a GoogleFS. All programs need to be modified to use the GoogleFS API.
Also, the GoogleFS has very narrow requirements/goals. It works best for programs that only append to the files.
-ItsME
1) ~$100 - nforce4 motherboard with 8 onboard stata,
/dev/sda5 to rebuild the array after a drive failure.
2) ~$40 - an additional PCI sata controller with 4 ports,
3) ~$100 - the cheapest AMD64 CPU you can buy, 12 400GB drives,
4) ~$150 - coolermaster stacker case
5) ~$1020 - 12 WD 400Gb drives
5) $0 - your favorite Linux distribution.
TOTAL: $1410
Each drive eats about 15W meaning around 180W with an additional 60W for motherboard/cpu consumption which makes it a comparable solution to an efficient scsi solution in terms of power consumption at a small fraction of the cost.
Personally, I created a raid1 array of 2 37GB 10krpm raptor drives for critical stuff and OS, and 2 raid5 arrays of 5 300GB drives for even superior cost per GB while increasing redundancy by a factor of 2. But that only gives you 2.4TB per mode in that case.
The configuration can be done with evms or lvm2, rebuilding on the fly and replacing drives on the fly should work just fine in theory (never tried on the fly), but if not, a scheduled 5 minute downtime is just fine also. My previous 0.5TB raid5 is up >3 years so far and a hard drive failure just required to mdadm md0 --add
Increasing the array size becomes tricky (although an available option) and fiddling with various distributed network filesystems doesnt really seems worth it for me personally, but openmosix and other clustering solutions offer distributed filesystems.
Just remember, the SATA architecture is nice, SCSI isnt really a requirement for this kind of solution.
http://www.terrascale.com/prod_e.html Run a client on linux boxes with user-mode drivers that provide a logical abstraction for a whole network of backend linux boxes over any networking transport you want.
I ran accross this a while back at linuxdevices it is supposed to scale to Petabytes and is the main technology used for the Internet Archive.
Capricorn Technologies Petabox
http://www.capricorn-tech.com/
Linux Devices Review
http://linuxdevices.com/news/NS2659179152.html