Slashdot Mirror
Building a Massive Single Volume Storage Solution?
An anonymous reader asks: "I've been asked to build a massive storage solution to scale from an initial threshold of 25TB to 1PB, primarily on commodity hardware and software. Based on my past experience and research, the commercial offerings for such a solution becomes cost prohibitive, and the budget for the solution is fairly small. Some the technologies that I've been scoping out are iSCSI, AoE and plain clustered/grid computers with JBOD (just a bunch of disks). Personally I'm more inclined on a grid cluster with 1GB interface where each node will have about 1-2TB of disk space and each node is based on a 'low' power consumption architecture. Next issue to tackle is finding a file system that could span across all the nodes and yet appear as a single volume to the application servers. At this point data redundancy is not a priority, however it will have to be addressed. My research has not yielded any viable open source alternative (unless Google releases GoogleFS) and I've researched into Lustre, xFS and PVFS. There some interesting commercial products such as the File Director from NeoPath Networks and a few others; however the cost is astronomical.
I would like to know if any Slashdot readers have any experience in build out such a solution? Any help/idea(s) would be greatly appreciated!"
register a few thousand gmail accounts and write the interface that will make writing of data to gmail inboxes invisible to the app.
Have you checked out GFS from RedHat (formerly Sistina)?
Can't you hook up 4x 7TB Xserve RAIDs to a PowerMac and use that?
Go check out veritas.com (now Symantec) for a comercially available filesystem...
Check out AFS.
UFS (universal file system) seems to be what you want here, any other thoughts?
Howabout the PetaBox, used by the Internet Archive ?
Livejournal developed their own distributed filesystem:
http://www.danga.com/mogilefs/
It's scalable and has nice reliability features, but is all userspace and doesn't have all the features/operations of a true POSIX filesystem, so it may not suit your needs.
I know a certain recent Zombie network that was discovered which collectively had quite a few Pbs of storage... Of course I wouldn't recommend going down that road as it leads to you know ... jail.
News Reporters Make Tasty Polar Bear Treats!
Check out the Internet Archive's Petabox. They have a 100 TB rack running in Europe right now.
call EMC. i am sure their clarion line will handle it.
i am unsure how you plan to do this with open source
software. It seems to me, you will want mgmt software
to go along with it. That is the real value, me thinks.
The Oracle Cluster filesystem is also available under the GPL. Dunno if that fits the bill; the description here is sort of vague. It sounds like a seriously ambitious project to approach for someone who doesn't even know what can be done, let alone what's within his budget.
Breakfast served all day!
Does not appear to be a single volume..
"Eve of Destruction", it's not just for old hippies anymore...
archive.org made a petabox
http://www.archive.org/web/petabox.php
There is now a company that seems to make the same design:
http://www.capricorn-tech.com/products.html
I don't know what FS they use, but apprently it is redudent.
May or may not be what you search. Quite expensive but impressive featurelist.
f tware/gpfs.html
http://www-03.ibm.com/servers/eserver/clusters/so
Meme of the day: I browse "Disable Sigs: Checked". So should you.
Check this out to do what you want.
This is the one of the coolest companies out there and their product is better than anything EMC has for storage.
http://www.falconstor.com/
-- Given enough time and money, Microsoft will eventualy invent UNIX.
What are you doing on a limited budget trying to build a 1PB solution? And why are you on a budget?
Just because you are starting at 25TB doesn't mean you aren't building a 1PB solution.
You also need to figure out what kind of bandwidth you need. It's very seldom that people have 1PB of data that is accessed by one person occasionally. If Some sort of USB or 1394 connection will work you are much better off than requiring infiniband.
Like many "ask Slashdot" questions this is the last place you should be looking for help...
LVM/Software RAID over Linux NBD.... ok it might suck, but I think it would work.
blah blah blah....
drightler@technicalogic.com
Considering that there are billion dollar companies whose only job it is to provide secure and redunant storage of the type that you describe, what makes you believe that someone on slash-dot would give you a solution for free?
The kind of thing you are talking about is non-trivial. If people have ideas concerning these matters you should pay them for them.
What a lot of gall!
Also, if you are being paid to do this by someone, then they obviously hired the wrong person to do the work.
My research has not yielded any viable open source alternative (unless Google releases GoogleFS)
Since when has Google released any open source software?
You might want to do a google search for "Linux Distributed Storage" or otherwise look at this old post on Slashdot which covered your question.
5 22247&tid=198&tid=230&tid=4&tid=106
Otherwise there are various solutions already availlable for free under Linux, but none will offer you a system that is easily implemented cheaply with fully redundant data storage.
http://ask.slashdot.org/article.pl?sid=05/05/04/1
Good luck,
-eks
If you know the scale of the problem, you should consult with a company like EMC to provide the support for this thing - you WILL need it.
Clustering the disks with iSCSI or ATAoE is trivial - you can do that very easily, but the filesystem to run on top of it is where you will have problems.
PVFS - has no redundancy - Lose one node lose them all.
GFS - does not scale well to those sizes or a large number of nodes - lots of hassle with the dlm.
GoogleFS - Essentially one write only - no small (50GB) files - little or no locking.
xFS - Way too easy to lose your data.
It seems that you only have one option:
Lustre - VERY Expensive - lots of hassle with meta-data servers and lock servers.
Go with a company to take care of all this hassle - you do not have the resources of Google to deal with this kind of thing yourself.
LL
Compete File System at http://www.python.org/pycon/2005/papers/46/Compete FileSystem.pdf.
MogileFS at http://www.danga.com/mogilefs/
I never thought I'd see the day when sites were boasting a petabyte of porn. .avis -- if you sat down and watched them end-to-end, you'd have 348 years of "backdoor sliders", "dribblers to short", "pop flies", and "long balls". We live in an enlightened age.
That's over 3 million hours of
Suppose each disk has a MTBF (mean time before failure) of 500,000 hours. That means that the average disk is expected to have a failure about every 57 years. Sounds good, right? Now, suppose you have 1000 disks. How long before the first one fails? Chances, are, not 57 years. If you assume that the failures are spread out evenly across time, a 1000-disk system will have a failure every 500 hours, or about every 3 weeks!
Now, of course the failures won't be spread out evenly, which makes this even trickier. A lot of your disks will be dead on arrival, or fail within the first few hundred hours. A lot will go for a long time without failure. The failure rates, in fact, will likely be fractal -- you'll have long periods without failures, or with few failures, and then a bunch of failures will occur in a short period of time, seemingly all at once.
You absolutely must plan on using some redundancy or erasure coding to store data on such a system. Some of the filesystems you mentioned do this. This allows the system to keep working under X number of failures. Redundancy/coding allows you to plan on scheduled maintanence, where you simply go in and swap out drives that have gone bad after the fact, rather than running around like a chicken with its head cut off every time a drive goes belly up.
I was going to suggest Reiser4 on LVM over a bunch of 4-disk RAID-5 arrays, but it seems that his definition or massive is more massive than mine.
NFS on Reiser4 on RAID-5 on AoE (multipath) on LVM on RAID-5?
What kind of availability do you need? Does all data need to be up all the time (like a bank/telco), or most of the data need to be up all the time (like google), or all the data need to be up most of the time (like a movie studio)?
...what your management was thinking. I mean, I can't imagine a storage requirement that large that you can build in a distributed model that would beat on price per GB an EMC or Hitachi or IBM or whomever SAN solution. The administration and DR costs alone for something like this would be astronomical. There just isn't really a way to do something this big on the cheap. I mean, this is what SANs were developed for in the first place. Its cheaper per GB than distributed local storage ever could be.
What?
We liked what we saw when we were looking for a similar thing. It's not cheap, but it's much cheaper than comparable stuff, and it runs well. We had an eval cluster and they worked like a champ.
15 zeros is no bytes at all... :)
Get your own free personal location tracker
This problem is a very real one in film production and we are moving in this direction for future productions after numerous faciliities got back to me with rave reviews of the speed, scalability and reliability of these units.
The nice thing is that they do scale, as the number of inodes grows so does the performance of the cluster and as you add storage you add bandwidth to your core filesystem. They are a great option for just this type of application.
The main issue you may run into at that size is that the real issue becomes having enough CPU horsepower to handle all the potential requests. This is where conventional network connected filesystem appliance solutions fall flat on their face. These seem to not have that issue at all. Only drawback is the obscene price Cisco charges for their infiniband switches they use as the backplane for the clusters. If you look at this as a potential solution you may want to pressure them to find another infiniband switch provider instead of paying the extortional pricing cisco's invented on their 48 port units.
Unfortunately, I should think needing a solution which can scale up to a Petabyte (!) of disk-space and a "fairly small" budget are at odds with one another.
Maybe you need to make a stronger case to someone that if such a mammoth storage system is required, it needs to be a higher priority item with better funding?
Heck, the loss of such large volumes of data would be devastating (I assume it's not your pr0n collection) to any organization. Buliding it on the cheap and having no backup (*)/redundancy systems would be just waiting to lose the whole thing.
(*) I truly have no idea how one backs up a petabyte
Lost at C:>. Found at C.
You may want to check out Panasas. From what I can discern, they are used by some high-end entities, but it does have the advantage of being high performance. You probably won't find anything this good for reading/writing out data from your app server to your data storage network.
BTW one of the founders wrote the original paper on RAID drives...
www.panasas.com
Check out the IBRIX Clustered Filesystem. http://www.ibrix.com/
the reason you can't find a cheap way to do this is because it just isn't cheap.
I would look at some lessons learned from Google. If you decide to go with some sort of homebrew solution based on a bunch of standard consumer disks you will run into other problems besides money. The more disks you have running, the more failures you will encounter. So any system you setup has to be able to have drives fail all day, and not require human intervention to stay up and running(unless you can get humans for cheap too).
Oracle endorses, uses Xserve RAID:
http://alienraid.org/article.php?story=oracle
http://www.apple.com/xserve/raid/
http://www.apple.com/xsan/
A buddy and I were talking of similar. We were looking for expandable large scale storage with good performance and cost and a high level facilitation of data management tasks (metadata management, media shuffling, accomodating technology advancements, etc). Decided on single nodes each controlling data storage to fit data use case. Each node presenting a span over ram, jbod, optical and tape in increasing size. RAM fs for that which is always in use, a couple TBs of jbod/ide raid holding more frequently used, everything in optical jukebox and safety backups to tape jukebox. Ideally the entire tower could be modular and auto-discovering, so when more long term storage is required a jukebox module could be added and the system could autodiscover capacities, alignment, etc. Data presentation done via a cascading hsm aware of each of the components and with optimized usecases (ie burning to blank optical when sufficient new data arrives on raid, maintaining indexing for each of the modules, etc). On top of the cascading storage would be a metadata vfs/presentation layer, to allow data navigation at a high level (cd /video/nature/banff/) via nfs or http web app over gig eth. A deligation/peering layer could allow for grouping such towers to grow in size. Tho quite theorhetical and the software being somewhat tricky, it could scale to the tens of TB/node size easily. We'd originally thought of an open hardware design with modules being added by the community and open hsm software supporting it. Neither of us has yet had the time to do much more than basic dev, but it's planned.
Why not just wait until those Atom Chip Laptops.
I mean, yeah the portibility is not what the customer wanted but the 6.8GHz CPU, cuppled with the 1TB of RAM should easily make up for the limited 2TB of HDD space.
Courtesy Link.
How about this: set up a bunch of mini-itx or similar low-power machines with 9 250GB PATA drives (and a single smaller drive for OS install), then use software raid under Linux to configure them as a single RAID-5 array (roughly 2TB) and set up an NFS server on the machine to share the array. Then set up a couple of controllers (for redundancy) that mount all the NFS shares and turn them into a linear or striped array(or RAID-1 for added data security). Then the controllers would be able to present a share using NFS, SMB, or whatever you need it to, that have a capacity that scales seamlessly... all you have to do is add more 2TB nodes to it. Obviously the details are flexible, like making the nodes RAID-1 instead of RAID-5 (and dropping them to 8 250GB drives for a round 1TB per node), but this should give you exactly what you're looking for. Your cost per node would simply be the mini-itx mobo, memory, 4 channel IDE controller card, and hard drives, and cost per controller would be practically any computer with high speed network interfaces (I'd recommend something with 2x Gb LAN at least, maybe Gb out and 10Gb to the HDD farm).
Look. Everyone wants a Lamborgini for the price of a Chevy. Cute. Yawn. Half of the Ask Slashdot questions are people who didn't find what they want at Walmart. Despite the amazing Slashdot advice, Ask Slashdot answers have somehow failed to put EMC, IBM, HP, etc. out of business. There is no free lunch.
Just call EMC, get a rep out, and give the paperwork to your boss. Do it today instead of 5 months from now and you will have a much better holiday season.
Note to moderators and other finger pointers: I did not say to BUY from EMC, I just said to show his boss how and why to do things the right way. It does not hurt to get quotes from the big vendors, mainly because the quote also comes with good, solid info that you can share with the PHBs. Despite what you think about "evil" tech sales persons and sales engineers, you actually can learn from them.
"If you want to improve, be content to be thought foolish and stupid." - Epictetus
You may want to take a look at IBRIX systems. They do a pretty robust parallel file system that has redundancy and failover.
That violates their terms of use pretty severely. I don't know what they would do (Google's not the "suing-for-the-hell-of-it" type), but that wouldn't last very long when they found out. And they would find out. +5 Interesting? Well, curiosity killed the cat.
Rank my idea: http://www.sinceslicedbread.com/node/531
And this is a little insane, but you MIGHT look at a combination of Oracle offerings. Specifically, Collaboration Suite for the end-user presentation (which gives you a web interface, and FTP/FTPS, and WebDAV, and they have a little desktop app that'll let you mount the WebDAV volume like a traditional SMB share, drive letter and all) and ASM for the disk management. ASM is an Oracle database (that can be a bunch of RAC instances for redundancy) that can take a bunch of disk, doesn't even need to be the same type or anything, just as long as you can present it to the ASM server somehow like NFS, and creates a sort of virtual data pool that could then be used for another, regular Oracle database like the one used as a datastore for the Collaboration Suite instance above.
Yes, I realize this is probably needlessly complicated, and since we don't have very specific information about what the disk is actually FOR it's also likely to be inappropriate for some other reason, but it could work, and Oracle Collaboration Suite (which is the only part you'd actually have to license, I believe, the rest just sort of comes with it) is licensed on a per-user basis. I'm not sure what the minimum number of users is, but for only the files-based part we're talking about here, I think the list price is something like $15/user.
UnionFS ought to do the trick.
I hope this is for facebook! Maybe to expand the new photo galleries?
check it out! have no experience with this company but looks very cool
perl -e 'print pack("H*", "6272616440766f74682e6e616d65")'
After reading that post and seeing the level of your english you should probably let someone else handle a project so complex.
blah, blah, blah
...can probably solve this problem for you. whether or not they can do so on the sort of budget you're willing to spend is a totally different story, however....
Coda works even when nodes disconnect, for instance with network outages or mobile computing. Plus, there is a Windows client, if that's the way your shop swings.
Anonymous marketingdroid?
Well, I don't know alot about this kind of thing, if XP doesn't have an upper drive size limit, could you just throw it in a big server case, thow in some scsi drives, stripe them (I *think* that's what it's called) and have it appear as one big volume?
I don't know what the limits of JFS are, but it sounds like a nice set up.
This article in Linux Journal ( http://www.linuxjournal.com/article/8149 ) talks about doing just that. The hardware costs ring up and don't scale as you get into your capacity ranges unless you can get a deal buying bulk HDDs - something like $10K per 7.5 terabytes
Exactly. This seems like somebody is trying to figure out a way to do something in-house which really ought to be left to either an outside contractor, or at least set up as a turnkey solution by a consultant. Given that he knows little enough about it that he's asking for help on Slashdot, I think this is yet another problem best solved using the telephone and a fat checkbook, and enough negotiating skills to convince management to pony up the cash up front instead of piddling it out over time on an in-house solution that's going to be a hole into which money and time are poured.
/. crowd for help, calling in professionals to take over for you isn't probably a bad idea.
I know people get tired of hearing "call IBM" as a solution to these questions, but in general if you have some massive IT infrastructure development task and are so lost on it that you're asking the
It's not even a question if whether you could do it in-house or not; given enough resources you probably could. It comes down to why you want to do something like this yourselves instead of finding people who do it all the time, week after week, for a living, telling them what you want, getting a price quote, and getting it done. Sure seems like a better way to go to me.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
If you're talking about building some sort of archival-type repository (like, keeping years worth of satellite imagery, for example), then you should probably look at the Centera from EMC. They scale into the petabyte range.
Providing you can find some sort of filesystem to support it (good luck), you could stash multiple arrays behind your host, or you could put in a TagmaStore from HDS with several arrays behind it. I'm not entirely sure how large the Tagma will scale, but the number 32 petabytes sticks in my head from a whitepaper somewhere.
I'd also question the perceived need to create one big filesystem to hold your whole petabyte of data. I'm a storage geek for a living, and I've found that usually after you start drilling into the application requirements, you find out that the app folks are either trying to use a data warehouse solution that's too small for the environment, or they're simply not aware of other alternatives available in their chosen app. No offense, but it sounds like you've had snowshoes strapped to your feet and directed to take a stroll through a minefield.
no really... http://www.redhat.com/software/rha/gfs/ from redhat seems to be what you might want.... take a look.
You've done your research so you may alreay know, but it's worth mentioning that EXT3 and ReiserFS will not cut it for your system. Most file systems (not to be confused with storage sub-systems) have a maximum volume size.
y stems
http://en.wikipedia.org/wiki/Comparison_of_file_s
Also - consider the problem of expanding any array.
As for how do do it, if I were to go super-cheap:
- Use software RAID-5 on each node (2 sub-systems, 8 disks each, hot-swap SATA 200GB, RAID-0 them and export as NBD)
- Use NBD to concatinate each node into a single block device
+ you DO NOT want to rebuild any parity info accross nodes with this poor-man's setup
- Use as many of GigE to interconnect with the nodes (limit use of switches)
Note:
- This setup will write data at multi 100MBit, but not GBit. It will read at close to GBit. I have this setup at home (1 node) and I'm impressed with software RAID and SATA.
- Contact the maintainers of any userland / kernel stuff you'll be needed and ask if they support the sizes you're looking for. I ran into trouble with dm-crypt (unsigned 32bit integer overflow) relating to file system size and mode of operation. All fixable.
I wrote a web application and a client in C# that uses gmail accounts as a sort of file system. using a set of email accounts as "index" accounts that use the gmail search functionality to find what you are looking for then pulling the attachment on the index to grab the parts of the file that where spread accross multiple gmail accounts in 500K chunks. it works really well. I did it for fun to see if I could. uses smtp to post the file chunks to a given set of accounts and users can donate accounts to the hive at will, increasing the overall storage size. all hosted maintained and index by gmal or any other free mail service as one big file system.
Hard disk space is doubling every 6 months - wait 5 years and you'll be able to buy a 25TB disk for $125.00.
A single raid50 of them will then give you your petabyte of storage, for around $6,000.
I have been searching for a solution for this as well. My current thought is that iSCSI is most appropiate. I plan to set up a number of small linux boxes, with as much storage space as a single system can accomidate, MD them so that each system is itself redundant. Each system will export an iSCSI target of the MD device. A single large node will then mount all the iSCSI devices and add anothe rlayer of raid (so that a single node failure doesn't result in down time), and export the file system as NFS to clients. I plan to just start with XFS for the on disk structures with an out-of-band journal.
Is http://www.hpss-collaboration.org/hpss/index.jsp something to try ?
The ECMWF uses this for their extremly large dataset. http://www.ecmwf.int/
OK first things first figure out the IO's you need to do and how they need to scale. If your looking for just bulk storage look into some nice big SATA drives. 4RU cases can get you 24 500 gig drives with 22 usable in raid 5 on a pair of 3ware 12 port or similar raid controlers or 11TB's per Unit. Serve these rater large chunks up with iSCSI. Take a HA cluster and merge those chunks together with software raid. The end servers just need to be fast enough to handle your interconnect speeds (gig or better I would hope) the HA pair needs a good deal of computaional ability to do raid calcs. All of them can use as much ram as you can shove in them if performance is a goal.
This isn't the fastest config by far but it's cheap and reliable.
Now with this being said there generaly isn't any good reason to make a disk that big, split things up if at all possible you do not want to deal with a PB of data in on huge volume.
No sir I dont like it.
Agami sells fairly large and very fast filers for cheap. Your 25Tb could be reached in under $170k with them. That's pretty cheap, in my experience.
“Common sense is not so common.” — Voltaire
From a Panasas press release:
A PETABYTE without redundancy? I can't imagine having that much data I didn't care about.
"A coward dies a thousand deaths, the brave but one."
register a few thousand gmail accounts and write the interface that will make writing of data to gmail inboxes invisible to the app.
Or you could use a system that posts to Slashdot. It will use signatures to ensure authenticity so Slashdot trolls can't mess up your data (or by Slashdot user, but I'd imagine they'd get banned pretty quick so you better use AC). To make changes to existing data it will write diffs (like CVS does). You will need a large number of IP addresses not in the same subnet (they can ban subnets). Getting past the lameness filter will be the only real challenge here, and that's not that hard.
Or you could go the easy way and use user journal postings to store the data.
#!/
http://www.coraid.com/products.htm
I haven't used it, but it caught my eye a while back and looks promising. 500GB per disk, 15 disks per 3U shelf, and up to 65,536 shelves per network means it's expandable from 7.5 TB for one shelf up to (theoretically) 480 PB or so.
You can wait for Sun to release ZFS, install Solaris 10 on an X86 box (or buy a new Sun X4100) Purchase as many Promise Vtrak 15200's as you require, configure them as iSCSI targets, and then use the Solaris 10 iSCSI initiator, and mount them. Then put them in your ZFS pool.
Use your head when configuring redundancy, and glory in your new found storage availability and capacity.
Good luck!
The company I work for uses LeftHand (http://www.lefthandnetworks.com./ They are an iSCSI solution. Totally scaleable, you want more storage, just buy a few more units. You can also carve it up to be one gigantic volume if you wish. Pretty darn cheap also. Beats the heck out of EMC on price.
Wny not just a bunch of PC's, each with 6x400GB drives? That's 2.4 TB per PC. 25TB is only 10 PC's -- 60 drives. What's the big deal just using NFS? Seems like not a very difficult target to hit.
Now, going to 1 PB -- that would be 400 PC's. At this point you've left the domain of something which is all that simple. Even so, if you use wake-on-LAN, you could no doubt get away with having 400 PCs, without special power, heating/cooling, etc -- as long as you were able to control the flow of information, so that no more than tiny fraction of the PCs would be on at a given time. And, of course, you'd have significant latency -- waiting for a node to wake. Since I don't know your requirements, I can't say how significant a barrier this would be.
You would get into a reliability issue with 400 PC's. Again, it would be useful to understand something about your problem space. It's possible that a butt-simple method would work for you.
Really, the difficulty/cost of the implementation is a direct function of data flow. If your data flow requirements are tiny, then the solution could be quite cheap.
Of course, even with low outgoing data flow, you'll still have your hands full just filling up the disks in the first place! Consider this: a typical cheap hard drive on a cheap PC can sustain, say, 5MB per second. If one PC is handling 2.4 TB, that would take 5-6 days to fill up. I suppose it would be reasonable to fill up, say, 10-20 PCs at a time. For the 25 TB system, you could fill it up in a week, easy. But for the 1 PB system, filling 20 PCs at a time, it would take you 3 months! Still, that might be OK, depending on your requirements...
Have you looked into the GFS from Redhat - or something like the Archivas (http://www.archivas.com/ ArC system - the latter is commercial, but sounds like it fits the bill for doing what you need, and it supports TPOF configurations, commodity hardware, etc.
Use a RAID.
(I'm not entirely sure what that is, but I know that's its storage-related, and this is Slashdot, and there's a small chance that even if my comment isn't at all helpful, it might just be incredibly funny)
One thing came to mind when reading this:
;)
:)
http://ohlssonvox.8k.com/fdd_raid.htm
Cheap hardware, commodity interface and storage media, dirt cheap... Now, you'd need over 18 million of the things for the low-end capacity, but they'd be easily replaceable, probably hot-swappable, and might actually be somewhat durable
I'd pay good money to get a tour of a company with rows and rows of iMacs with 127 floppy drives hanging off each one...
Go have a chat with your local EMC, HDS, HP or IBM rep .. this is bread and butter stuff for them and they'll give you heaps of whitepapers.
... but I'm sure you've a valid reason for doing it .. right ?
Personally I think you'd be insane to put it all in as one volume and then sharing it to multiple computers
Software you might want to look at :
Transoft FibreNet (gobbled up by HP, may still exist as one of their products)
Tivoli SANergy (not bad actually)
Only time I've ever needed anything this massive is for my porn collection which is only in the terabytes, not petabytes, sorry.
Want to find other gamers to play board and role playing game
They need you as much as you need them!
s lip/
http://www.theregister.co.uk/2005/10/25/sun_grid_
The three choices are:
1) Big
2) Cheap
3) Easy
You've already chosen Big, I'd recommend making your other choice Option 3. As a couple of other people have said call EMC, Hitachi or IBM and price a SAN. After you've got your hardware, drop a copy of Veritas VxFS/VxVM on your server(s). The first time you'll find out about a hard disk failure is when the wrench monkey from whichever hardware vendor you decide on shows up to replace the disk because the box called for help and Veritas makes FS/Volume management so easy even a paper MCSE can puzzle it out.
I disagree with what you say, but I'll defend your right to say it to the death - Voltaire
There isn't enough detail here at all to begin with recommendations.
Small budget...
There's small ($10k) and there's small ($1M). Small? Quantify that.
What is the nature of the data?
There is a massive difference between a PB of static files that are rarely accessed or indexed and a PB that is a highly transactional database.
What size are the files themselves?
What SLA has to be met with regard to the throughput?
Most of the arguments already made in favor of redundancy are also true. How could you have a PB of anything that isn't worth protecting? Just the cost of assembling a PB of data or having to restore it from a single disk fault would pretty much demand having a volume management system in place with redundancy.
If none of this has occured to you, hire someone with the background needed to architect this for you.
1. Price out 25 TB of storage using 250 GB drives.
2. Multiply by 4 to account for media failure
3. Multiply by 5 to account for support hardware
4. Multiply by 2 for support and maintenance
5. Multiply by 2 for unexpected changes
6. Note that thiss solution will NOT SCALE to 1PB and multiply by 10
7. Compare numbers with EMC/TERAData/IBM/etc
End up buying the turnkey solution anyway
Food not Bombs is a nice platitude but it breaks down when you notice that the Bombees are usually well fed
Check it out...fits the bill nicely.
I would shy away from software... There are Raid controllers out there that will span a raid across multiple controllers, and multiple machines. LSI has some nice ones.
You can put together a ton of NO-O/S servers with the raid controllers, and interlink them all doing a hardware raid across the machines. Only one machine {preferable the main node} needs an O/S and then you can attach it to the network through that O/S.
It costs more, but is FAR more reliable.
Sadly, he's right. It's funny to see marketing guys clamoring about like worker ants from rival colonies going after the same crumb, the crumb in this instance being the contract for storing our data. But in the end, it's a hell of a lot stress-free and *perhaps* cheaper in the long run to get an all-in-one solution like EMC. Unless you've got a very talented and devoted group to create some brillant software solutions, that is.
The eternal struggle of good vs. evil begins within one's self.
Sounds VERY strange to me. We only have 2TB of storage and its raid 10.
To answer your question.. you get what you pay for. The last sysadmin here was a lets do it on the cheap kinda guy, and it took me 2 years to fix that shit. The way you save the cash is you play vendors off each other. I just bought a fully redundant high speed storage array that does everything for 82k. (list price is around 180k) Took a few weeks, and I played vendors off each other like mad puppies, but I got more than I wanted for what I wanted to pay.
If you try to do storage on the cheap, 75 tb of storage for what I'm paying for 2 is cheap, so if your budget is under 150k or so, then look for another job. You are never going to have anything but problems.
Everything on one drive is a huge problem too... seek time will be out of this world.
Just three more hours seapeople and you can finally take me away from this crappy God Damned planet full of hippies
As a VERY satisfied customer, I say, just buy the damned thing from EMC. There's few enough warm fuzzy feelings that SysAdmins have in this day and age, like your CE calling at 7:00am saying: "Hey, you had a few hard SCSI errors on Disk 3 Enclosure 0 Tray 0 last night, that's your production LUNs isn't it? There should be a courier there with a disk by 10, and I'll stop by to make sure things are hotsparing back properly after you replace the disk okay?" And *THIS* is just because my CE knows I can handle replacing a disk. Normally he'd come out and do that, and sit around while it re-built the Raid Group.
Yeah, EMC costs. THIS is why. The support, when needed, is top top top notch. Which would you rather have in a DR situation?
What if it is just turtles all the way down?
http://www.novell.com/products/openenterpriseserve r/iscsi.html
NSS 3.0 does up to 8TB I believe. XFS does 9PB?
Service guarantees Citizenship! Questions Guarantee GITMO.... Amerika Uber Alles!
If designing for speed, NOT cost:
given 2PB = 1 Human Brain, non interlaced
1024 TB == 1 PB
1 TB == 1 PC Computer with 1200GB H/D, 2Gig RAM, Networking
If designing for cost, NOT speed:
1 DVD = 4.5GB
1 PB = 1024 TB = 1,048,576 GB
1 PC Computer, with a DVD like the one mementioned above.
1 Robotic CNC Arm, with DVD Gripper(tm)
1 Very Huge Wire Cage to hold DVD's like a Juke Box.
(This has been done before, but with Tapes)
See http://lucene.apache.org/nutch/ and look for Nutch NDFS (something similar to Google's FS you mentioned). I use Nutch over at Simpy (think Web 2.0) and am very happy with it.
Simpy
It seems like I found a magic disk on my system that will take unlimited data! I've been storing huge amounts of stuff there for the past few months, and still haven't run out of disk space. Try it out: /dev/null
Retrieving the data, on the other hand, has been problematic... I'm figuring that when I really need it, I'll just post to Ask Slashdot and somebody will help me out.
I'm sorry, but if you're going to build a massive disk system like this and plan on filling a good portion of it up then you need to have backup plans in place. In the event that you lose data, how will you recover?
So this isn't about how to build 1 PB of disk space but two: One is the main and the other is the backup.
Can you afford that?
Can you afford to lose the data?
Read the post that you're replying to more carefully next time.
There seems to be lots of SATA-RAID based iSCSI SAN devices available nowadays.. Some links to products I have seen:
t m
t m
h tm
:)
d uct_detail/dataframe_420.html t _53700.html?view=1&curNodeId=0
/ section/Product~Categories/category/iSCSI/options/ IPBank/drivetype/L~Series/formfactor/Integrated/in face/SATA~-~Serial~ATA
o rmat.jhtml?articleID=170702726
http://www.equallogic.com./ They make nice SATA-raid based iSCSI SAN devices with all the features you could expect (volumes, snapshots, array/volume-expansion, hotswap, redundant controllers, redundant fans, etc).
http://www.equallogic.com/pages/products_PS100E.h
14 250G sata disks, 3U, 3.5 TB of raw storage.
http://www.equallogic.com/pages/products_PS300E.h
14 500G sata disks, 3U, 7 TB of raw storage.
http://www.equallogic.com/pages/products_PS2400E.
56+ TB
Looks good. I have not yet used them myself
Another iSCSI SATA SAN possibility:
http://www.mpccorp.com/smallbiz/store/servers/pro
16 sata disks, review:
http://www.infoworld.com/MPC_DataFrame_420/produc
This company also has SATA iSCSI SAN devices:
http://www.dynamicnetworkfactory.com/products.asp
iSCSI SAN comparison:
http://www.networkcomputing.com/story/singlePageF
There are also software iSCSI target solutions for use with your own/custom hardware.
http://iscsitarget.sourceforge.net/ for building linux-based iSCSI target/SAN.
If you are familiar with iSCSI targets / iSCSI SAN devices please post your comments!
Why would you need one single volume for 25 TB of pr0n?
Or is there anything else that eats up these amounts of space?
On se Internetz nobody noes your German.
but I'm just a linux hobbyist and programmer, so take any advice I give with a grain of salt, but here's what I did for my setup at home. To start, you're looking a little over $1000 per TB. And, that's about as cheap as it gets with redudundancy. I have 8 drives in one machine, it's in a RAID 5 config, and I have a hot spare. However, if I were doing this for a mission critical application, I would have it in a RAID 6 configuration with a hot spare, and buy a hot swap cage, which would further add to the costs. Then, I would simply export the RAID 5 volume using ISCSI, and then see if there is a way to RAID all of the ISCSI volumes using a master server. I imagine that if you do it right, you could scale up such a system to a fairly large number of machines. You would probably want something faster than gigabit eithernet, probably 10,000 MB/s connecting everything together, otherwise, things could get a bit congested at the head node.
Where all this could get terribly expensive is in power requirements, it requires less power to run a cage of hard drives than it does to run a network of PC's. I'd imagine that any money you save on hardware, you would spend on your power bill. Either way, your looking at, bare minimum, about $30K to start for 25TB's, and I would add another 10K padding just to be safe, to pay for stuff like UPS (which you want), a high end switch (which you'll also need), cabling, etc. In other words, it's not cheap, and like my parent just said, it will probably be cheaper in the long run to have someone like IBM do it for you. Do you really want to be responsible for 25-1000 TB's of data?
Although ZFS is not immediately available, it should be before long. Though this does not address your hardware concerns, choosing hardware compatible with either Solaris 10 or OpenSolaris would be beneficial, in my opinion.
A good ZFS introduction.
I am amazed, this topic just screamed for one...
... ...
In Soviet Russian, Beowulf cluster jokes make you!
Ugh I feel so dirty.
MoM++ - A Classic Expanded - [Master of Magic 1.5]
http://mompp.sourceforge.net/
You are being asked to do the impossible. The simple answer is get more money and get something like a NetApp solution that has complete redundancy and protection.
There are so many issues that you are completely unaware of:
1) Data redundancy. 25 TB is a huge amount of data and a hell of a lot of disk drives. 1 PB is more than you can imagine. What is the likelihood that 1 of your disks will fail? What about 2 disks failing? If you lose two disks, you will lose everything, can you handle that risk? Netapp has RAID DP which, unlike RAID 6, offers protection from 2 disks going down, without the performance hits of RAID 6.
2) How are you going to allocate the space?
3) What about when you add more space? How will you handle this?
4) What are the physical limitations that you will encounter?
5) What about network card redundancy? Can your company handle the downtime if a network card fails and you need to go in a exchange it?
6) What about power failures?
7) Off-site storage?
8) How are you going to manage all these drives and be aware of what their status is?
9) What about when volumes get full?
The problem is that you have spent so much time thinking about technology when technology isn't the main issue, its how to handle something this large.
Do yourself a favor and go with a reputable vendor like NetApp and save yourself a whole bunch of trouble.
If you really wanted to be cheap, just NFS mount that cluster(fuck?) to a Linux box and do all your user management from there. That way you'll only use 1 connection to the Netware box. Download a copy of Netware and check it out.
"I can't give you a brain, so I'll give you a diploma" - The Great Oz (blatently stolen sig)
These are getting monotonous.
I want a terabyte storage solution, I want a networking solution, I want a cheap computer solution.
Want.Want.Want. Sounds like Bill Connolly's 'I want' rant.
Where are your specs!! and by the way the answer is 42.
Duh, With two petabytes. ;)
Web Developers: Celebrate to our roots! Animated Gifs and Tiled Backgrounds, dont let our history die!
I think Oracle plans to release the second version of its cluster filesystem (CFS2?) as stand-alone, no Oracle DB needed. I guess it's not ready for primetime yet, but it might be interesting in a year.
You might also want to consider that the number of files on a petabyte-sized file system will not play nicely with things like Windows Explorer or various backup programs. But before all that, I'd check that the requirement that you have a file system this big is tied to a project that will make money for your business. I think the request for this implementation is based on some unsound logic -- e.g. how are you going to index all that? -- and should be reviewed before you go off half-cocked and start implementing.
If the logic were actually sound and I really thought this were a good idea (I don't), I'd talk to EMC/Hitachi/IBM just so you can get a price tag on what actually implementing this will cost. Then cobble together your homebrew solution and ask the company to pay you the difference in cash. Step 2: move offshore.
Check out MogileFS http://www.danga.com/mogilefs/ . It is open source and might meet your requirements.
Storage nodes: 7 x 2.8TB 2U RAID5+1 boxen with Serial ATA. The 2.8TB is logical, not physical. The OS for each of those machines is RAMDISK based (something we concocted based on what I read about the DNALounge awhile back) so it helps curb disk failures of the storage nodes themselves. We avoid disk failure by using RAID5. Of course that doesn't protect against mutiple simultaneous disk failure, but read on for more. Each of the storage nodes is exported via NBD.
Then we have a head unit, a 64-bit machine. This machine does a software RAID5 across the storage nodes using an NBD client. Essentially each storage node is a "disk" and the head unit binds and manages the sofware raid5. So let's say a whole storage node goes down (for whatever reason it does), all the data is still intact. RAID5 rebuild time over the gigabit network is about 18hrs, which is acceptable. We even have another storage box as a hot-spare.
On top of that, we have the whole cluster mirrored to another identical cluster via DRBD in a different geographic location. This is linked by Gigabit WAN. So if we have a massive disaster and lose the entire primary cluster, then we have a 2ndary cluster ready to go. We needed to purchase the Enterprise version of DRBD ($2k US) but that's worth it because they're neato guys.
We use XFS as the filesystem. This system gives us 14TB of redundant "RAID-55 with a Mirror" space. Both clusters together? $85k.
When the cluster starts running out of space (about 70% or so), we add ANOTHER cluster of similar stats to the initial one and use LVM to join the two units together.
This has scaled us to 30TB and we're pretty happy with it. The read speed is very good (hdparm says Timing buffered disk reads: 200 MB in 3.01 seconds = 66.49 MB/sec) and the write speed is about 32 MB/sec. For what our application is doing, that's a fine speed.
The filesystems going to be the hardest component of this. I know of no open-source fs that could handle this. I'm assuming this is all online storage, and there is no desire to nearline it to tape. Ideally, you'd want something that could contcatenate multiple LUNs (of RAIDed storage) without having to run through a volume manager. Nothing agaist volume managers, but it'd be another component to support. Looking at proprietary FSs, you've got CXFS from SGI, which could easily handle the PB requirement and plays nice on Linux. Sun's got QFS, which would max out at 1PB and could do the volume management bit easily. Linux support was a little flakey last time I used it, but it's a free download and evaluation, you could go get it right now.
IBM's SAN-FS would also meet the capacity needs and would have the advantage of providing nearline capability, if you're into that. Sun's SAM-FS is basically the QFS product with nearline-to-tape capability. Linux is only supported as a client OS there. Of course, if you buy the mantra that Solaris is 'open-source,' then that might not be an issue.
As for hardware with any of the above solutions, you're going to be looking at using multiple RAIDing disk enclosures of some kind. At a budget, probably SATA disks talking to the controller, and iSCSI to the host. FibreChannel to the host would be a little more costly, but might be worth it since iSCSI is just getting mature enough to be usable in production.
Dear Slashdot,
I have been tasked with (insert very difficult, very important job). This is very important to my company. I have (insert number much lower than it should be) dollars to do this. I do not want to use (insert company name specializing in this exact thing) because management thinks they are too expensive. I think I can do this (insert better/faster/cheaper/...) than said company, even though they have vastly more experience and have invested much more time and research than I have. My continued and future employment probably rests on this project. Please advise.
Maybe not
...of a porn collection do you have?
It's distributed I believe.
Capricorn Tech? They power the Internet Archive. "Capricorn Technologies was founded in 2004 and provides petabyte-class storage solutions for organizations worldwide. Capricorn's PetaBox technology grew out of a search for high density, low cost, low power storage systems for the world's largest data collections. Capricorn Technologies is proud to be a leader in the next data storage revolution."
...because what good is power unless you can abuse it?
As always these questions lack an actual budge figure, even a ballpark number would be nice. Not to mention the specification of the data being laid down on the disk.
members are seeing something, your seeing an ad
I've been setting up linux based SAN (used to be NAS) for a few years now, and I know for a fact you can make a 25TB~100TB SAN with commodity hardware and low end drives...
There are many ways...
-LVM over Serveral 5-10TB RAID5/6 servers (through FC, iSCSI or AoE)
-Software RAID 5/6 over Serveral 5-10TB RAID5 servers (through FC, iSCSI or AoE)
This is the cheapest yet still reliable way to work with low end hardware. Expect your servers to crash, your disks to fail... AoE is not CPU intensive and doesn't require expensive HBA, this is a good solution if you want to build a custom/noone will never understand/ cheap SAN. iSCSI is easy to deploy but very CPU intensive (without HBA). FC is extremly expensive.
Anyway, with this kind of setup, you'll only have "Mass" Storage, performance will be bad, security as well and you'll probably have a very high & frequent system downtime...
I'd be more tempted to just not worry about the power supply issue and go whole hog. My company recently purchased a NAS device called a Terastorus from Aberdeen solutions. Thing runs like a champ. Comes with an 80 gb internal HD for Storage Server 2003 and 24 500 GB HDs. We raid 5'd it and added a hot spare at the total cost of two terabytes. Total cost: maybe 7 grand.
But it's a heck of a lot better in both performance and cost than our EMC AX-100. That's just a big turkey!!!
Of course it weighs like 75 lbs and cranks out heat like a banshee, but it was cheaper than our two TB AX-100 and is a heck of a lot more reliable!
Nexsan has a box called ATA Beast
Raid, Fibre Channel, 42 ATA drives per 7 RU chasis. Throw in 500GB drives and 1 parity drive for every 6 data drives and you have ~30 TB per chasis.
Fly me to the moon Let me sing among those stars Let me see what spring is like On jupiter and mars
I've been asked to build a massive storage solution to scale from an initial threshold of 25TB to 1PB, primarily on commodity hardware and software... ...At this point data redundancy is not a priority, however it will have to be addressed.
What you're asked to do isn't always what needs to be done. You're making a huge mistake if data redundancy for this enormous project is just an afterthought.
I don't know what role you play in your organization, but try to get the business-minded folks to tell you what they want to accomplish, and then YOU and your architecture people will decide what needs to be done to accomplish it.
With such vague requirements, how can they already know that you should build it from scratch instead of choosing a turnkey solution?
$8.95/mo web hosting
I can't think of any specific versions, but if you don't need performance, a database-driven filesystem that sits on top of a networked file system should be doable, but costly.
A toy version:
DB simply keeps track of filenames and reasonably-sized chuncks of files. Chunks of files are spread far and wide across the LAN.
The local filesystems store each chunk as a local file.
Hardware RAID provides protection against disk outages.
Performance sucks, and like any fake file system file semantics may not equal a true local file system.
Per node hardware configuration:
Fast server stuffed with let's say 5 SCSI raid cards, dual power supplies, each with 15-disk RAID5 using 0.5TB drives with redundant power supplies: well under $30,000 for 7TB.
Power consumption: 3KW I'm guessing - hopefully that's a huge overestimate.
1PB is 140 of these give or take. That's $4.2M and 420KW of power.
Even if the power is 1/10th that, 42KW is still a big power bill.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
I appreciate this might not seem like helpful advice, but...
If you've been asked to do something this by a company that can afford to buy one commercial off-the-shelf high volume storage solutions, then I honestly can't imagine any solution they try and knock up will actually work (as I'm not aware of any free software solution that's currently up to the task).
If your company doesn't have / can't raise the capital to buy a commercial system for a project of this scale, I can't possibly see how they could afford to screw up on this and go with an untested idea that could very well end up being a huge money sink they wouldn't be able to dig themselves out of - one that could doom the entire company and all it's investors given the cost it could run to.
And of course, for such a big project, they should hire people who would already know how to do something like this (which is not a dig, it's just crazy to skimp on staff when you have an ambitious project which requires large amounts of capital investment).
That said...
I were going to do large scale storage on the cheap, depending on the design of the software and the specific requirements (particularly if I was also developing the software we were going to use, or was able to set feature requirements and/or was able to make the modifications myself) I would build the largest standard file shares I could with SATA disks (using commodity hardware, hot swappable, running linux, with front loading drive bays).
The specifics of handling the load balancing (via multiple front ends, multiple mount points, pre-deteremined hashing to balance things out, proxies/caches, hooks in the file system calls, hooks in the application to talk to a controller, etc) depend entirely on the sort of application however.
It's definately likely to be far easier (and more cost effective) to have the software take care of knowing where the data is stored, rather than trying to build a single really large file share. I know at least one very known large company who've went down this route (with essentially elaborately hacked up versions of common OS software).
The downside is you have to support whatever hack you come up with to do this, but that shouldn't be an enormous amount of work (and you can probably afford to hire someone to support it full time for significantly less than the cost of a support contract for a commercial solution).
The amount of space that your talking about needing its going to save you TIME (and remember TIME=MONEY^2 in the IT field), Money and Headaches. You will Never build anything that scales into that kind of Territory without replicating the kind of technology that EMC(or any other storage company, however EMC is the clear leader in the field a little more $ upfront, but HUGE ROI over the length of the product) has already created. The ease of use of the product and the tools available for it are going to save you and your company money in the end. Its a little more upfront, but the money you will save over the next 5+ years in labor costs, hardware maintainace, and POWER costs is worth it. A huge cluster full of JBOD is going to chew electrons like nobodies business. Never mind your time building and maintaining it...I will bet you that it will never be fully stable day to day, some part of it is always going to need poking at.
Power Corrupts,Absolute Power Corrupts Absolutely, leaving one person(group)in charge is absolutely corrupt.
Same with disk drives — most failures will be clustered around the 57-year mark. Not that your attitude towards redunancy is wrong. Just as people sometimes die in infancy, some disk drives break down quickly. So there's a chance that you'll lose some drives from your thousand-disk system in the first year.
How big a chance? To answer that question, you need more statistics about drive failure — and a much better grasp of probability theory.
What's making your question hard is the "make it like one volume" restriction. The problem is trivial otherwise. If I were you, I'd be asking whoever tasked you with this to *really* justify on a technical level why they need it to appear as a single volume, since that makes all the possible solutions slower, more costly, and more difficult to maintain.
Chances are extremely high that what they really want is a "/bigfatfs" directory visible everywhere in which they will store many discrete items in subdirectories by project or by dataset or by user. You should convince them to let you build it from commodity machines serving a few TB each mounted as seperate filesystems underneath that umbrella directory. Then your only challenge is coherent management of the namespace of mountpoints for consistency across the environment (which there are longstanding tools for, like autofs + (ldap, nis, nis+, whatever)), and administration/assignment of new space requests within your cluster (that could be scripted to automatically allocate from the least-used volume which can satisfy the request (where least used could mean space or could mean activity hotness based on the metrics you're logging)).
11*43+456^2
Storage with AtaOverEthernet. The cheapest Midium size storage...
And write a fs wrapper to acces clusterd jdbc proyect...
And mysql as file repository?
Ata Over Ethernet
http://freshmeat.net/projects/aoelinux/
Ata Over Ethernet tools
http://freshmeat.net/projects/aoetools/
c-jbdc
http://c-jdbc.objectweb.org/
Mysql
http://mysql.org/
Â_Â
Get a box with four 250GB IDE drives and find an old copy of Stacker! 1TB should become 25TB after installation.
Click here or here.
Specs: 10u rackspace, 1500w power, $115k cash (medium selection + 35 BYO drives)
XServe RAID and XSan.
http://www.apple.com/xserve/raid/
http://www.apple.com/xsan/
'nuff said.
Perhaps - and I don't know your requirements - NFS will work, with sub-servers for each portion.
The real problem here is you want commodity hardware - but really 99.999% uptime that
expensive hardware gives you. Otherwise any solution you come up with will have reliablity
problems - you don't want those problems with a system that big.
Think about it - you are talking about hundreds of commodity IDE drives.... all working
in unision - one goes down they all go down. Unless you have a raid-array controller.
NFS - sort of solves that - in effect - NFS servers come and go... and [can, if properly
configured] be automatically re-mounted [an auto-mounter]
This would also require that you segment your data into segments - perhaps you can do that
or - perhaps you cannot. This also assumes that (a) no one file is huge, (b)you do not
require "hard-links" across sub-sections (c) speed is not totally critical, but capacity is.
There you have it: Distributed storage that scales by simply adding more computers, monkeys and floppy disks.
The folks at the Internet Archive have already done the hard work of figuring out how to create a petabyte storage system using commodity hardware. The system works so well they started a company to sell PetaBoxes to others. Why reinvent the wheel?
What once required talent and brilliance today only requires reading a how-to file, configuring,
and rebooting.
EMC is obsolete. Their customers just haven't discovered it yet.
-I like my women like I like my tea: green-
Dude, you just told us you come in three minutes.
This guy's full of crap, and it's further testimony that slashdot's editors are also full of it.
No one, and I mean, no one, builds multi-petabyte storage silos without actually understanding storage. It's obvious from the questions in this posting that this guy is so far out of his league that he should quit hist job and go live on a beach in a cardboard box somewhere or he's another dumbass trying to sound intelligent.
Thanks for speaking and removing all doubt.
Did I get a troll rating????? Did I? Cool.
Democrats and Republicans are like AIDS and Cancer, I want neither!
there's no free lunch. everybody wants a lamborghini for the cost of an impala. good, fast or cheap -- pick two. nobody puts symmetrix/shark blah blah blah out of business with duct tape and bailing wire. Jesus. if you're all so bored, give up the keyboards and get real jobs. :)
:) not free, but then again, not as ludicrously expensive as the EMC/Hitachi/IBM/NetApp alternatives.
anyhow, the heads over at archive.org spun out a company to develop storage systems closely matching the brief you just laid down. Check this out: Petabox
you can buy the nodes and their stuff is proven in the archive.org infrastructure.
http://www.lefthandnetworks.com/ supports all that of what the person is talking about in the article. As you add more of these units, the volumes are spread over the units you add. This means that you can add storage as you go and still have redundancy. You can configure each individual unit to use RAID 0, 1, or 5, and still get to have a volume, or many, across multiple storage units that in turn have parts of a whole voule or set of volumes. Its like haveing double mirroring, once within each individual storage unit level (which has many IDE drives in RAID 1, or 5) and then twice at the storage unit level. Of course this assumes that you have at least two storage units. And, yes, this means that to have redundancy you ahve to add them in pairs (I think) and have some storage units in one physical location and the pairs of each of those in another location for disaster recovery (fire, earthquackes, you know things can happen.)
I have worked with this units and they kick ass. You can do snapshots of entire servers quickly, given that you have the right infrastructure, set thresholds for voulmes that can be increased or reduced on the fly, brick level restoration of files!!!, etc. And of course, my respect goes to their engineers. I saw them working on one unit cause we had a really bad power failure that killed one HD. Man those guys know their stuff up and down, and I've never seen anybody type commands so complex and so freaking long at that speed! They fixed the damn thing and got 99.99999% back from limbo!
I guess their storage boxes follow the model of LVM which is pretty cool and the storage boxes run Linux!!!
Don't take my word for it, go to their website and take a look 'cause I tend to confuse people with my posts rather than pass info efficiently.
Have a good one.
===== "Every head is a different world so don't invade mine you FREAK!" smartSAGA said
MatrixStore from Object Matrix http://www.object-matrix.com/ uses commodity hardware and clusters it together to create a highly expandable, reliable and secure storage environment.
How about TinyDisk?
;-)
-MJ
How about this:
- Use LVM on every node to make the 2TB seem like a single disk ( Assume 4 x 500GB disks )
- Use iSCSI/AoE to make the LVM volumes available on the network
- Use LVM again to merge exported volumes
- For redundancy use software raid 5 on the lvm volumes
I suspect there will be a lot of problems with efficiency but I think you should be relatively safe from hardware failures as the software raid will detect and repair them.
Anyone have any idea whether what i mention is possible/recomended?
The link I quickly googled examines the hardware side of your project and provides a little insight into the complexity of reliable, large-scale storage solutions. (http://ssrc.cse.ucsc.edu/Papers/hospodor-mss04.pd f) This could probably make a nice research project for a college or university if you have a large grant, but if you are talking about a business solution - I'd go with a commercial vendor with a proven track record and a verifiable list of satisfied customers. You don't want YOU to be the single point of failure if a petabyte of valuable data is lost, compromised, or even unavailable for any length of time. Downtime for a large database spells the loss of big bucks for most businesses and/or short employment for the responsible IT personnel.
I want to say it is 16 Tbyte offhand, but I'm not sure on that.
Short research indicates this was a limitation in 10.3, but I haven't found anything confirming or denying that 10.4 still has it.
Not that we've been looking into large amounts of Xsan storage here, but our requirements are a bit different. You can't hook >600 nodes up to the storage via fibre. Our problem is scaling out the NFS servers to be able to push all this data around.
Slashdot Patriotism: We Support our Dupes!
Stop what you are doing right now. If your architecture requires you to have one huge volume then you have architected things wrong. Imagine trying to fsck this damned thing! What about file system corruption- What the hell are you going to do when you lose a Petabyte of data because of some file system corruption? Small, sensible, easily managed smaller partitions are the way to go. Use a database to organize where given files are stored. Do something that makes sense. I have a client now who just lost a bunch of data because they used a system like this.
Having said all this- If you are still intent on finding a good file system then use AFS. It's probably your best free solution. If you want to sleep at night call EMC.
-sirket
GPFS
Take it from someone who's messed with nearly every storage product on the market, if you want something that works fairly simply, performs at approaching spindle speed ( meaning the file system is not the bottleneck - if you have 10 GB/sec. storage bandwidth, expect to see near that with proper tuning ), is very stable ( compared to most storage solutions on the market - bear in mind that most storage products are aimed at large-block sequential I/O, and fall down - either performance-wise or stability-wise - when you throw other I/O patterns or combinations of patterns at them ), and is portable across nearly any Linux distribution ( with varying amounts of difficulty, I have had to hack their kernel patches before when using a unsupported kernel ), GPFS is the one. Of course, the problem there is I believe it's pretty expensive to run on non-IBM hardware. But if you have IBM hardware ( even if it's not the hardware you're running the FS on ) or some sort of in with IBM, they'll let you have it for a song and a dance.
Having said that, Lustre is getting there. I'd say it's the equal of GPFS ( as a parallel filesystem - I believe it is even more flexible as a distributed filesystem ) in performance, probably scales roughly the same ( haven't played with it in a large installation, so can't tell you beyond looking at the architecture ), and is going to the be the biggest player on the market in the future. It's also free ( IIRC Cluster File Systems sells support, but the code is freely available ) and not tied to IBM and whatnot, like GPFS is. Of course, HP has a big connection with Lustre, but not ownership thereof.
Those are really the only two that I would consider for a serious high-performance storage project. If you don't need great performance, that's when you can start looking at things like GFS, ADIC's StorNext, Ibrix, etc.
Oh, Gautham Sastri ( of former Maximum Throughput fame ) has a newer company called Terrascale, I recall them putting on a presentation at the 2003 or 2004 ( can't remember ) Supercomputing conference ( SC2005 is coming up in a few weeks, yeah!!! ) which showed pretty good performance ( relative to the small system they were using ), not sure how they're coming along...
Anyways, good luck...and don't forget to use Iozone to benchmark the damn thing!
PC moderators can suck my White pierced, tattooed dick. If you think pride == hate, s/dick/Aryan meat mallet/g.
Approximately 764 million floppies would do the trick, give or take a few, although you'd probably want some kind of ultra-efficient volume catalog system...I'd say a good Jet Database configuration could work.
Why not store the data randomly in a dilithium matrix with asynchronous data transfer and AJAX? Maybe some RUBY on RAILS too - I hear that's hot right now. Of course, you'd have to make use of a couple of Heisenberg compensators configured in parallel to keep track account for any memory addressing issues, but no need to state the obvious there.
"Nokia is not a country, it's the capital of Finland!" -Moderated "Informative". Yeesh.
We at Vap-o-tech 2003 Inc. (not associated with Vap-o-tech 2001 Inc. which has closed its doors due to allegations of investor fraud) have developed ToastFS 2003. Using patented CRUMB technology and high capacity BUTTER read/write caching, we are able to turn your average loaf of Wunderbread into a 200gb storage media. Simply buy a loaf of our own specially tested Wunderbread ($250 USD) along with a USB-to-Popup Toaster interface (don't worry, USB 2.0 is more than capable of handling 120amp wall sockets without a problem, except in California). Then take our Vap-o-bake ToastFS drive and pop two pieces in. For doubled capacity, buy our Vap-o-bake ToastFSx2 drive, which takes four pieces. From a command prompt, simply type FORMAT C: and answer yes. Your new ToastFS drive will be formatted in minutes. Please note that we have 24 hour technical support via 1-900-842-8524 ext 241. Please don't hang up. Our operators in the Dutch Antilles are very busy and could take upwards of an hour to get to you.
The world's burning. Moped Jesus spotted on I50. Details at 11.
<shamelessplug> As a customer, the 3par solution has been very impressive to me and the company I work for. We have EMC arrays, Netapps, etc, but the 3par blows them all away in performance/size/just about every aspect including price, and we are currently migrating as much as we can off the other solutions onto the 3par. To make it more flexable (the unit itself is designed for fiberchannel), we got a set of onStor NAS gateways, and they make NFS actually faster than local disk (using Gig-e). The 3par is highly modular, and the software to use it makes it simple to reconfigure the volumes/raid type/whatever. It also does snapshots for you.
Tm
Support TBI Research: http://www.raisinhope.org
You could get a bunch of Broadcom 8 port SATA controllers, which equals about 4TB per controller. 4 or 5 controllers = 16-20TB per box, then you can run the cables into an outside drive bay enclosure and one box can control 40 500GB hard drives.
If you're not doing any processing on this, a good CPU should be able to handle the load.
If you need web hosting, you could do worse than here
I manage a small (29 dual-xeon nodes) linux cluster in a lab for my local college. A while ago I had the same problem when we ran out of storage space on the main file server.
/dev/md# blcok device;
/dev/md0 device;
/dev/md0 array, and I was done.
My solution was to use the nodes' hard disks (each one has a 120GB Ultra320 10000rpm disk) combined in a network RAID1+0 solution (we use gigabit ethernet) to get more space. With that aproach you can get as much redudancy as you need.
Heres what I did:
1. After install the network block device server (nbd-server)in each one of the nodes, I created a 100GB partition on the HD and exported then directly using the raw mode;
2. On the master node (using the nbd-client) I created a block device for each one of the nodes partitions;
3. After that I installed the linux software raid tools (mdadm) and created a small RAID1 array for each pair of nodes. I ended up with 14 100GB network RAID1 arrays each one with its very own
4. I created a big 1.4TB (14 * 100GB) RAID0 array with the 14 RAID1 ones and attached it to the
5. The final step was to create a large RaiseFS filesystem on the
You have to pay special attention to the array shutdown and startup procedures. I wrote my own scripts to take care of that for me.
Our array may seens small compared to what you are looking for, but I am pretty sure that it will scale well for arrays much larger then ours.
Good luck.
SAN
Scalable, fault tolerant, etc...
1/5 boxes arrived DOA. The ethernet cards didn't work. The cables to the hard drives weren't long enough. The hot-pluggable disk trays were flakey. The BIOS had to be flashed. The properitary hard drive controller drivers sucked, had to buy new controllers. 1/10 disk drives were DOA.
Three monhs later, and $40K poorer, we had a system that couldn't pass 24 hours of stress testing without failing in some wacky way. For the $40K and my salary time, we could have bought a usable system from IBM or HP or whomever, and it would have worked. Engineering big systems is non-trivial.
http://www.isilon.com/
You are being screwed.
10% of failures is completely unnaceptable, i can only assume yur solution is crap or you are making the numbers up.
Dear Poster.
This is Slashdot India and Slashdot China. With over a Billion people combined we can do it by using PeopleRAID, which is hot-swappable once every generation, and redundant once we eliminate our "one child per a couple" restrictions, and increase our Viagra imports.
What your looking for is the goolge file system. It uses desktop grade hardrives to support larger than pentabyte storage requirements with fast file access. This year at Sigmetrics 2005 the inved talk was
Google - Or how I learned to love Terabytes
Urs Hoelzle, VP of Operations and Engineering, Google Inc
He certainly made it sound like the google file system was available for use.
You were given a task that can't be done cheap or simple. The best path to success IMO with the specs you described, is to pick up the phone and call IBM or SUN and say "Help!". Chances are you will end up with something where the cost per gig is going to be very close to what you would pay had you done it yourself, but without all of the wasted time and wrong choices.
Want to save lots of money? Go lean on the support by learning what you have, how to take care of it, and SUPPORT the system yourself.
Just for kicks, call up a sales rep at Iomega and let then know you want to do this with ZIP disks. Record it and blog the MP3 of their head exploding.
Good luck.
The real catch to your problem seems to be the single volume issue.
Would be fairly easy to network a bunch of boxes that added up to the requisite storage amount... but accessed as a single drive with redundancy would be an issue. As mentioned the number of disks invovled pretty much means you can't go into it without be worried about redundancy unless you can deal with data loss pretty much from day one.
If you have to do it on your own my guess is you are going to have to take the largest easily available solution and then do your own work to scale it up by gluing those together. And there are any number of ways to do that... I'd suggest tackling it with the tools you are most comfortable with... or if your not up for that level of development it is time to tell those who must be obeyed that they can't get there from here if they want a non proffesional HUGE single volume storgae solution.
I don't ask you to be me. I only ask you not expect me to be you.
There's a reason why Terabyte storage arrays for commercial applications cost a lot of money, and why consulting services from IBM, EMC, Hitachi, etc. have the huge per-hour cost. If you/your management can't see that, you really have no business being there. Sure, anyone can throw a JBOD RAID together for a thousand bucks, but I wouldn't trust anything more important than MP3s to it.
ACHTUNG! Das computermachine ist nicht fuer gefingerpoken und mittengrabben. Ist nicht fuer gewerken bei das dumpkopfen.
Their founding engineers are old-school storage guys who built a new system from scratch in 2002. The SAN supports FC and iSCSI front end, FC and SATA backend and natively virtualizes all storage into a single pool. When you want more storage, just add more controllers, enclosures, etc.. - the array supports n-way dynamic controllers. They don't have a ton of information on their website because they sell entirely through partners. The demo is worth your time b/c you'll see where SANs are headed.
dCache
In short, your stated objective smells. Not enough data.
WHAT is going to be done (database, file storage?)
HOW will it be accessed? (One large file, many smaller files)
WHEN will it be accessed? (During business hours, distributed over the day?)
AVERAGE TRANSFERS - will the whole schmear come over, selected parts?
SECURITY a concern? (Sensitive data, protected network)
BACKUP - a petabyte of tape storage is expensive, and takes quite a while to do.
POWER - do you have enough?
COOLING - ditto
SPACE - ditto - my $DAYJOB computer room is about 3000 sq ft... and we're going to be using all of it within 12 months.
That said, if you go with big drives over a lot of systems, use lots-o-nics to keep the nic from being the bottleneck. A single gig connection sounds fine, but wait until you have 100's of people going for files at once. It'll get swamped. And swear off V-SAN from Cisco. Not worth it at all.
Necessity is the plea for every infringement of human freedom. It is the argument of tyrants; it is the creed of slaves.
Why not have those 7 boxes all export their storage via GFS and create one large volume like that?
Somehow every time an article like this gets posted, the poster forgets to mention - OBTW, my budget is only .....
If you can afford a 25TB sized storage system, that will scale by a factor of 40, and still be recoverable, and you are asking slashdot for suggestions . . . really just give up and call EMC or their ilk and be prepared to write some zeros in your check.
OTOH it can be done - 6 years ago I interviewed at Up and Coming Photo Site that planned on archiving every photo uploaded by every user for all time for free. They had standardized on a really cheap and no doubt hideously dodgy open source software raid-5 white box design that they could churn out for a few kilobucks, and were building them by the hundreds and connecting them with NFS. With SATA RAID-5 controllers, ReiserFS, and the Linux VM in much better shape now than then, a project like that might actually be fun!
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
Just a comment about MTBF. It's often not understood, and it is one of my little pet peaves with tech producers because they don't try to correct it. MTBF is a rating for reliability to achieve lasting the warrenty period.
You have a drive that is rated 500,000 hours MTBF. Suppose you bought a drive and let it run at rated duty. Driver are normally rated to run 100% of the time, but many other devices will have duty period. Further, you run the drive until its warrenty is up. You then throw this perfectly working drive out the window and replace it. If you keep the up this pattern, then approximately once per 500,000 hours on average you should have a drive fail before the warrenty period is up. This is why it is important to not only look at the MTBF but also its warrenty period.
As a side note: In theory, you should be throwing drives out on a periodic basic. One way around this is to not buy all the same drive type and manufacturer. By having a pool of drive types, you distribute, thus minimize, risk of drive failures. Additionally, you may want to have a standard period of time for drive replacement so as to shedule your down time, as opposed to it all being unexpected.
Bel, the mostly sane.. "Of course I can't see anything! I'm standing on the shoulders of idiots." -- Me
The best way to have access to have all that idea is to have a parallel filesystem. There are many type of parallel filessytems. Some are designed for certain types of files others for other sorts.
PVFS and GPFS can be said to be the two most useful type of filesystems to be in use. I would reccomend that you use somethng like attached storage through fibre or through a SCSI because if a part of the array goes down then you can jump start it by replacing the SCSI array or the storage node.
Make sure your data storage drives are raid 5 so if you have one disk that goes bad then you don't have to call the whole file system to do a recreate.
Making a large filesystem is expensive. But go with the parallel filesystem option you will see how fast and easy it is. And also try to use some 64 bit system so you can have larger ammounts of metadata etc..
And PS with such a large filesystem say bye bye to the inode problem.
Sad. That post had more jargon than a 1999s dot-com press release.
It looks like marketing-speak has crossed over to the everyday vocabulary of the Slashdot crowd. Didn't a "massive single storage solution" used to be called a "hard drive bank?"
Aren't you the people from our local university in Gr.n.ng.n building that giant distributed satelite?
Quisque verborum suorum optimus interpres...
Why not just using the best solution -- Coda !?!
You know, Mom & Pop will probably be happy once you go out of there, despite your wet dreams.
It is the year 2015. I have a petabyte in my iPod.
the FS can cope, but it does insist on redundancy
Funny. It's this the same quandry that those "old" computers faced. . .100,000 tubes with a MTBF of 1000 hours so one would fail every 5 minutes. Not having been there I can't say from first hand experience but I read it someplace. . .
It may sound crazy but the cheapest solution with relatively low power consumption and a high reduncancy I can think of would be some bizarely large PC (Quad Opteron w 16 GB RAM or something) and 30 to 50 external USB harddrives attached. Add in Linux and some virtual Software RAID thingy set up to make good use of the Horsepower and the only problem you have left might be IO speed. Ext3 is a slowpoke, but it's free, stable and safe - and probably fast enough.
The biggest problem is finding USB adapters that can handle the load and enough sockets to plug them into. I don't know the USB specs, but from what I can tell USB 2 is far more powerfull than people usually expect. You'll need a little scripting to keep track of all those drives and their state, but it should work on the software side.
Your power consumption would be extremely low for 'homebrew' and redundancy and inexpensiveness would be best. And you can get everything for that at your local PC shop. Exept the Quad Opteron Board maybe.
Let's see:
Biiiiiig PC + 5 heavyweight USB 2 Cards => 9000$
60 external 0,4 TB USB HDDs => 18000$
12 USB Switches => 1000$
Backup HDDs, USB stuff and spare parts => 5000$
Sum: 33000$
That's extremely cheap. If it works with all those
USB drives hooked to one Box this is your ticket.
Crazy but feasable none the less.
We suffer more in our imagination than in reality. - Seneca
If you want is just SIOS (Single IO Space) then a PVFS2 setup with multiple dataservers would be the way to go. As long as you do not care for concurrent writes to the same data then PVFS2 would be the easist and cheapes solution to go. You can even setup simple stripe so your writes will be distributed (round rubin manner) amoung the avalible data server. Note however that parallel file systems such as PVFS2 are distinguised from the more general term: distributed file systems, in that they are designed for multiple clients accessing the filesystem data in parallel from a pool of machines - so if what you really want is just to have one machine access a large storage pool then the lack of meta data cahcing in PVFS2 may prove to high a price to accept in which case you may want to give GFS a try.
"Politics is for the moment, equations are forever" -Albert Einstein
The PetaBox, as previously discussed on Slashdot sounds like just what you want...
Make a phone call to those guyes http://www.archive.org/
Better go practice... 3 minutes... hmm...
Yeah, I'd like three Ferrari's and my budget is $26.00
What kind of crap Ask Slashdot question is this????
Go spend the $x Million required for a handful of Hitachi XP1024's and stop looking to Slashdot to provide your cheap-ass low-budget IT "solutions".
You might also check out the Internet Backplane Protocol, or "IBP", which was designed to store massive amounts of data in a generic "cloud".
For instance, more than 18 months ago, it was already moving 1TB per week on Internet2, and this past week was at 1.896TB.
There are a number of commercial vendors out there which have very interesting but not necessarily cheap solutions. Have a look at iBrix and iSilon.
I am currently in the process of deploying a Sun QFS/SAMFS based filesystem. This environment is being built out to manage about 300TB of data. QFS is a shared, SAN based filesystem, that as far as I am aware, has some kind of obscene maximum filesystem / name space size
The latest version of QFS (4.4 I think) supports Solaris 10 x86 as a full peer which would allow you to install on commodity servers, switches and disk (I'm using a McData switch with IBM and Nexsan disk) for a ~$5000 a box software license, each of which could export the shared QFS over NFS/Samba/FTP whatever for non Solaris clients.
Sun ships a Linux client, but so far it's kind of crappy.
There's been a lot of discussion about whether the closed source modules gpfs requires violate the kernel GPL license. But who would be crazy enough to sue IBM?
Yeah, I bumped into them at the Broadcast show in Amsterdam a while back, we are looking into their stuff in detail to archive all the video we have on tape right now. Not free but they are being pretty agressive with the pricing right now.
Well, at least the development and test process would be fun.
. html
And based on what I have heard, most SATA raid-5 controllers are not quite ready for prime time, although I did recently interview with one outfit that was running their whole enterprise on Pogo Linux StorageWare boxes:
http://pogolinux.com/storage/sata/storagewaresata
I dunno. I would want to pound the crap out of them for a few months before I committed. Worse, in my last job we had zero budget and a bunch of ancient DL380s with hardware raid; nothing bad ever happened but it kept me awake at night, mostly thinking of new scripts I had to install to make sure every box was at least rsync'ed somewhere else. Now, I'm fortunate to work at a place that can afford EMC and Sun, SCSI and FC stuff. Nothing ever breaks, ever, and I sleep like a baby.
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
Hell, you could just buy several miles of ethernet cable and keep all the bits moving in the network. Imagine how fast retrieval would be if the data was already being transmitted before you requested it? Make sure you use Cat6 though, don't skimp on data integrity!
--
"Outlook not so good." That magic 8-ball knows everything! I'll ask about Exchange Server next.
How are you going to backup that monster?
You said you do not need data redundancy, since backup is nearly impossible, how are you going to survive a disk crash (they *will* crash!)
2bits.com, Inc: Drupal, WordPress, and LAMP performance tuning.
I work for EMC. If anyone needs a petabyte of storage and is willing to buy it all at once, I can get you under the price mentioned above. It will be our largest, slowest drives with minimal host connnectivity, but it does include 7+1 RAID protection. If you are serious, drop me a line. Yes, a saleman will call.
Nothing for 6-digit uids?
Use openfiler. It is good.
In the past, our management splurged and bought a Network Appliance. Two, actually; we have an F880 and an F740. The 740 is pretty much defunct right now; it's only got about 300GB of disk and we use it to house static application installs. The 880 is more robust and therefore more efficiently used, but only has 2.2TB of (insanely reliable) disk. Disk that costs about $20,000 per additional TB.
Last year, we learned that Xyratex (the company that makes the disk shelves for NetApp) has started selling SATA-based disk arrays. Right now, I believe they only support 400GB SATA drives in a 16-drive chassis, but support for 500GB drives is supposedly right around the corner. A fully-populated chassis with 400GB drives will yield about 4.8TB of usable space. We have purchased five head units (about $20,000 full of disk) and one shelf (about $15,000 full of disk). Each unit is expandable to 7 shelves (including the head), which yields over 32TB of usable disk. I don't know what your budget is, but $110,000 is pretty reasonable for over 32TB. Admittedly, you could buy 32 stripped-down Dell Dimension 4700's (to get SATA) with two 500GB hard drives; install a slim OS and you could get approximately the same amount of usable space. But the reliability of the Xyratex has been far greater than the reliability of the Dell machines we've purchased in recent years.
It's ironic that you brought up the NeoPath File Director. We're going through the trials and tribulations of installing a clustered pair of them right now. We've had some difficulty in getting them set up, but it seems like they'll do the trick when we get them going. The MSRP on the cluster is kind of high, but talk to their sales guys if you're interested - we got almost 1/3 off the listed price. We plan on using the File Director to migrate old files from the NetApps to our Xyratex, thereby expanding our storage at $3,300 per TB instead of $20,000 per TB. I can see it working well, though, for aggregating a large number of file servers into a single virtual server.
I don't know how the File Director will interface with your operating systems. We use Veritas's Storage Foundation (~$500 per license - you should only need one) on our systems because we're primarily limited to Windows and Solaris, which have difficulty with large filesystems. Storage Foundation breaks the size limitations as well as enabling easier management of your volumes.
I hope this helps. Good luck.
pr0n, even i think a tb is enough...
This is a trivial task if you have a few key tools.
One (1) management/virtualization box/head.
Six (6) 5TB iscsi targets.
Why six, because RAID is nice and all hardware fails regardless.
In the first box, you need to import all six targets for raw storage. Using a robust raid/scsi stack to glue all the segments togather should yield a raw 30TB base under RAID 0, or 25TB base under RAID 5.
Now to provision the bucket of storage.
I am sure this is boring and trivial, so I will stop.
Whom ever you are feel free to email me offline if more details are desired or skip it and read the next message.
Cheers,
Andre
Look at this..... http://www.archive.org/web/petabox.php
I've created enormous filesystems (500TB and up) with the following recipie:
As many ATABeasts as you can afford (they'll currently do about 20TB each)...
As many QLogic 2GB fibrechannel cards as you have ATABeasts
1 or more Sun V880s (Just keep buying them as you fill PCI slots)
Veritas Volume Manager/Veritas Filesystem 4.1 or higher.
NFS/Samba to your heart's content
Now, I realize that there are two glaring problems with the above recipie. The first is that only Samba is open source, and I understand theres something of an issue about slashdotters using software that they actually have to pay for. Fear not--in this case the result will be software that actually works.
The second is that Sun will only look at 2TB LUNS and no bigger---and Veritas will only see ~1 TB "drives" or smaller. This actually isn't a problem--simply configure your ATABeasts intelligently, throw a fuckload of LUNS at Solaris (It can take it) and smoosh them all together into one enormo-volume with Veritas.
The SC|05 (SuperComputing) conference in November, will have some tracks on high performance storage capabilities as well as a special initiative called StorCloud of which the goal is to build a petabyte scale storage farm on the show floor (http://sc05.supercomp.org/initiatives/storcloud.p hp) A lot of the major storage vendors will be at this conference too.
While not all of this may fit in with your organization's goals and requirements exactly, you might find it a useful forum just to check out what these folks are up to and take an opportunity to chat with them.
Just a thought. Good luck!
A better solution for petabytes of storage is a commercial product developed by some former Cray engineers, SAM-FS. This is really an HSM solution -- very scalable, which is the entire point.
... disclaimer: I don't sell SAM-FS or Solaris. This is just a good, scalable solution that I've used at a large government facility and can be bought OTS from Sun or StorageTek for fairly serious bucks.
The samfs filesystem allows you to browse to a file as if it's online in a disk even if samfs has cached it to some other media, such as DVDs or tapes. The nice thing about this HSM style solution is that it combines completeness (all files available, even if they are on tape and rarely used) with the need for data integrity. SAM-FS allows duplicate tapes or dupe writes to DVD or CD-ROM; it can even write the duplicates in another city. Why not? You've can have your 2 PB in NYC and your backup 2 PB in LA. I think that's a worthwhile feature.
The hitch for some folks here is that SAM-FS is written for Solaris only, so far as I know. It is darned fast for what it does, although retrievals of rarely-used files tend to be limited by the HSM media type(s) used in the particular storage system.
Hmm
This is totally out of my ass, but wouldnt mind finding out if its possbile/feasable. Could the person setup a bunch of standard PCs each stuffed with disks, run software iscsi servers on those pcs to share each disk. Have one main master server using software or hardware iSCSI add the disks to an Logical Volume. When more storage is needed, a new iSCSI serving pc could be added to the volume. Im not sure what file system would work, but then again, I am not sure the idea would work at all.
You dont know what your talking about.
The fact that you would call ext3 stable and safe is a dead give away, the ext file system
family is an abomination when it comes to saftey.
>(*) I truly have no idea how one backs up a petabyte
Okay, you know how those flash keystick thingies are starting to look really cheap? You just get about....
This is a hacked account, for which the owner can not be held responsible.
File systems asside, your talking about a whole lot of hardware here! Is it really necessary to have all this data online at the same time, is it possible to store it in some other way (ie tapes) because it would probably be a whole lot cheaper!
Well, lets see... using 300Gb SCSI disks (assuming you can find raid 0 hardware to support enough disks) you can build out a 1Pb storage system with about 3,334 disks. That would set you back about $3.3Million, assuming you paid retail prices for the disks ~$1,000 / disk @ CDW today. Of course, if you orderded 3,000+ disks, I'm sure they would cut you a deal on the price.
Any hope of daisy chaining together a few dozen direct attached storage devices to a NAS server? Something like a Dell PowerVault 220 with 14 300Gb SCSI drives will set you back about $21K and give you 3.4Tb / 3U of space (RAID-5) so you would have some saftey net built in (albeit not much). Slap 10 of these on a Powervault 6000 series and you should have a ball park of 34Tb (while shy of what your looking for gets you in the right direction). Total cost around $250K - do it four times and spread the work out over four logical volumes and you should get in the neighborhood of 1Petabyte. You could then set up a redundant server structure and for $2Million you have a redundant mirrored architecture ready for one to fail and be brought up online quickly.
AF-Design, web development.
More info on SAM.... SAM-FS and SAM-QFS are both closed source commercial products developed by LSC which was bought by SUN a few years back. At work recently I had a chance to test SAM-QFS, in my lab. To do a shared SAM-FS(QFS) file system the metadata server must be a Solaris machine. Clients of the file system can be Linux, I forget which kernel versions are currently supported. The shared file system part of SAM-QFS scales better than RedHat GFS (largest file system = 8TB) by supporting 252 disk luns per filesystem @ 2TB/disk lun. The HSM archive part of SAM-QFS seemed to have a few issues with very large #'s of files > 40 million, such as file system backup time and containerizing MANY small files into a container to archive. Also SAM-QFS isn't a journaled file system...there wasn't any consistency check done on the file system dump / restore utilities. I could do a dump of the file system, change a block in the dump file, restore the dump, and have corrupted data -- in my case I hex editted the names of file names and restored it and it didn't complain one bit. I'd be interested in what other people might suggest for a good archive/HSM product. I've looked at ADIC AMASS, ADIC StorNext, FileTek StorHouse RFS, SUN SAM-QFS, HPSS, IBM SAN FS and haven't found one that was good enough and scalable to the 1 billion files/FS and 1+ PB of data level....
It sounds like your requirements are good (scalable) , fast (or large in this case) and cheap. I'm pretty sure that only 2 out of 3 are possible.
/
/. knows that :)
How important is the data? More importantly how much will it cost if you lose the data? Without redundancy if you lose one disk you are likely to lose the whole volume... and cheap disks WILL die. If your budget won't allow you to purchase a real enterprise storage system will it allow you to buy an adequate backup solution?
Check this out it might be your best bet http://www-03.ibm.com/servers/storage/disk/ds4000
I just sat through a demo of one of these units. I was impressed with their goodness, storage capabilities and the ability to use SATA drives brings the cost for large amounts storage cheaper than your typical SCSI enterprise storage solution. These boxes are also fully RAID capable... but you still need a backup... but everyone who reads
Build a system based on the Broadcom RAIDCore BC4852 controller.- Office/Storage-Solutions
x .html
http://www.broadcom.com/products/Enterprise-Small
Tomshardware sucessfully ran 32 S-ATA drives in a single box in RAID5 mode (2x 16 drive array because of software limitations). With the current generation of 500GB drives that should yield you close to 15TB. Add several more boxes along with some clustering software.
http://www.tomshardware.com/storage/20041006/inde
In a motherboard with 4x PCI-X slots you will get very good performance for your money. S-ATA drives may not be as reliable as SCSI, but they (along with the controllers) are cheap so you can always have a few spare around in case a drive died (you would with SCSI anyway). Linux drivers are available ofcourse.
We have a few redbox racks running in SF now. The Archive grows by about 25TB or 30TB a month, and all of our new storage is Petabox racks of redboxes. We are also retiring some of our aging whitebox systems and replacing them with redboxes, copying their contents over onto the newer media (and md5-doublechecking the contents before we let go of the old box).
-- TTK
Check out www.coraid.com - they use ATA commands over Ethernet for a cheap and scalable storage solution.
Do you really think a petabyte is a lot of space?
:-), I mean even boxes with 30 drives in them have a facilities impact at home.)
Yes. At your rate, assuming you don't accelerate your [movie downloading, film-less video production/processing, simulation/model data acquisition, account/ticketing for your million-client monthly expansion, vhosting aggregation, etc.]:
It will take you two months to fill a terabyte... so 2000 months to fill a petabyte. If you keep buying new drives every couple months it will only take you 166 years to reach your petabyte!
(This ignores practical problems
"Honey, I'm going to need a new edition to the house to hold my 1000 hard-drives."*
"Um, no."
(*) Assumes storage densities improve a little over the next 150 years
Read Heinlein's 1953 Revolt in 2100, now more than ever.
I was interviewed for a job there:
http://www.exanet.com/
Toms hardware has an artickle about the 70 TB online backupsystem of University of Tübingen
x .html
http://www.tomshardware.com/storage/20030425/inde
Just do like they do on scrapheap challenge; send a couple of maniacs to a junk yard and let them build a diskarray from for example parts from an old Dell powerserver and a desktop case. Although the circuit card may be too large, so you may have to cut them of. Example/tutoirial (although in swedish:
s to2000/
http://www.acc.umu.se/images/archive/20050517-Pla
I need a jet that goes 1000mph. I looked at the commercial offerings, but they are price-prohibitive. For the time being I don't need landing gear. I was hoping that the Slashdot folks could help me.
I think you're waiting for ZFS, the last word in filesystems. Interesting stuff here.
Google FS is not a real file system. It's just a bunch of API's that the program calls. The GoogleFS is not integrated into the Linux VFS. So you can't mount a GoogleFS. All programs need to be modified to use the GoogleFS API.
Also, the GoogleFS has very narrow requirements/goals. It works best for programs that only append to the files.
-ItsME
1) ~$100 - nforce4 motherboard with 8 onboard stata,
/dev/sda5 to rebuild the array after a drive failure.
2) ~$40 - an additional PCI sata controller with 4 ports,
3) ~$100 - the cheapest AMD64 CPU you can buy, 12 400GB drives,
4) ~$150 - coolermaster stacker case
5) ~$1020 - 12 WD 400Gb drives
5) $0 - your favorite Linux distribution.
TOTAL: $1410
Each drive eats about 15W meaning around 180W with an additional 60W for motherboard/cpu consumption which makes it a comparable solution to an efficient scsi solution in terms of power consumption at a small fraction of the cost.
Personally, I created a raid1 array of 2 37GB 10krpm raptor drives for critical stuff and OS, and 2 raid5 arrays of 5 300GB drives for even superior cost per GB while increasing redundancy by a factor of 2. But that only gives you 2.4TB per mode in that case.
The configuration can be done with evms or lvm2, rebuilding on the fly and replacing drives on the fly should work just fine in theory (never tried on the fly), but if not, a scheduled 5 minute downtime is just fine also. My previous 0.5TB raid5 is up >3 years so far and a hard drive failure just required to mdadm md0 --add
Increasing the array size becomes tricky (although an available option) and fiddling with various distributed network filesystems doesnt really seems worth it for me personally, but openmosix and other clustering solutions offer distributed filesystems.
Just remember, the SATA architecture is nice, SCSI isnt really a requirement for this kind of solution.
http://www.terrascale.com/prod_e.html Run a client on linux boxes with user-mode drivers that provide a logical abstraction for a whole network of backend linux boxes over any networking transport you want.
I ran accross this a while back at linuxdevices it is supposed to scale to Petabytes and is the main technology used for the Internet Archive.
Capricorn Technologies Petabox
http://www.capricorn-tech.com/
Linux Devices Review
http://linuxdevices.com/news/NS2659179152.html
That is what I currently have -- a cluster with 4 servers and about 10TB of distributed data... Now what about the single name-space thing. The only way I have been able to solve the name-space thing is to aoutomount the file servers and then symbolic link various directory trees... Not very desirable.
Suggestions?
Great! Please build me a 100TB storage array with 128GB of cache in your garage. Not shared cache, please allocate it intelligently per logical device I create. Also, please make sure I have 64 front end fibre channel ports so I can attach this storage to my server farm. Oh, and also have it dial your house when a drive fails and be at my site within 4 hours to replace it.
I may want to connect my mainframe to it as well. While you're at it, build me a second one so I can synchronously or asynchronously mirror logical devices to my datacenter across town. I may want to do cloning and snapshotting to make copies of my production database, so throw in those capabilites too.
Please let me know when it's ready so I can drop my obsolete storage vendor.
I cannot speak for the poster, but when I played with this problem myself I spent *weeks* trying to get AFS and Coda running on a modern 2.6 kernel... just for starters. I thank the wonderful person for the GFS link (I'll try that too). Instead of just critisizing can you post a link or two yourself?
Nerd TV did a show on the archive org founder: Brewster Kahle.
He talked about doing storage on the cheap.
Here is a link to the system they are using in production.
This solution is GPL'ed. It also appears you can buy it as well.
Just out of curiosity... How long does it take to watch 25 TB of porn?
Why worry? Each of us is wearing an unlicensed "nucular" accelerator on his back.
Sig changed for readability by G.W.
Sure, it quite possibly *can* be done. In the same way that you could theoretically build a spaceshuttle from a T-Ford and lots of old spraycans!p 12000/index.html or even http://www.sun.com/storage/highend/9990/index.xml.
When you're aiming for extreme solutions, you have to use quite extreme components. Think http://www.sgi.com/products/storage/ http://h18006.www1.hp.com/products/storageworks/x
The only imaginable way to get petabyte on commodity hardware that I can think of is to build a seriously huge Beowulf cluster. But putting a singular FS on about ~500 seperate computers in a cluster is rather madness...
Take the good advice many fellow slashdotters has made, do NOT use a singular filesystem that spans 1 PetaByte or more...
http://oceanstore.cs.berkeley.edu/
At least, they have pond.
I know they're late to the party, but the FAS3000 line supports the new shelves with SATA drives. This drops their cost down dramatically (over 50% reduction), with the same unusually high NetApp standards. We're upgrading our F820c cluster with FAS3050c heads, and adding some SATA storage for new projects. I'd have to agree with the parent that NetApp is definitely a good solution, though none of their individual filer heads, or even two head clusters, can scale this far up. Their single volume limitation is still at 17.6TB.
Pardon me for asking a dumb question, but why the fsck do you need all of this in a *single* volume? I can understand the need for single volumes, and the need for large volumes, and even the need for single large volumes up to a point. But your reqest is taxing my understanding.
A Government Is a Body of People, Usually Notably Ungoverned
I realize that I will most likely never be seen by the other Anonymous Coward but would like to add a possibility. I know that some laboratory in this country was faced with the same problem and eventually had this company SDSNM do the work. Their web page is scant on details but I vaguely recall that specification required 100MB/s for a year using off the shelf hard drives. Worth contacting. If anything happens because of this mention LB.
try filetek.com, its a massive solution but comes with a far cheaper bill at the end of the month, startup may be steep, but it looks like a hard drive to a windows box just like that 100gb one sitting in your computer.
You might want to check out xrootd if this is read-only data. It does work for read-write but it isn't as performant as you might want without serious application work. This is a server that uses a redirector to send clients to the machine with the actual data. The web
site is http://xrootd.slac.stanford.edu/.
It's in the admin guide buried a ways down into it in a table of other limits. You'd think they'd have that on the web page, a petabyte is a big number!
Slashdot Patriotism: We Support our Dupes!