Slashdot Mirror
RAID Solutions For Terrabyte Databases?
gullevek asks: "We are about to implement a huge database. In the first two years it will grow up to 650 GB and it will get larger and larger afterwards and could grow up to 2-4 TB. I never before implemented such a large DB. The DB Software has been choosen but now I have to find the right hardware. The basic components are not a problem, but what about storage? I would prefer to use RAID, of course, but what type of RAID? RAID 5 may be the best for disk failure, but it can be quite slow. RAID1 is fastest but the most expensive. Especially when it is at this sizes.
And what about the type of drive: SCSI-3? FCA? FibreChannel? Do you folks at Slashdot have any suggestions?"
Folks like EMC make their living selling, servicing, upgrading, and guaranteeing uptime for big systems like this.
Put the word out that you need a some RAID, and a service contract to go with it. Start talking with other folks who have similar sized solutions, and see who their vendor is.
Unless you are using MySQL, your DB vendor probably has some handy tips on how to handle disks -- how many to have, how to configure them, what to allocate each one or group to...
I don't see a question in this unless you're using MySQL or some other relatively "low-end" DB in which case, you probably have larger concerns to deal with.
-JF
MrJoy.com -- Because coding is FUN!
Yeah, I didn't say it'd be cheap. But if they're going to do this, you may as well pay someone who's got engineers who fantasize about data striping and mirroring.
As usual, rant first then opinion.
When I see questions like this on Slashdot, I get chills. This is obviously a big-budget job and yet the guy responsible for the project seems to be asking some very basic -- too basic -- questions. Honestly, this isn't a flame. I've been in that boat myself from time to time. However, before I'd every consider holding myself up to public ridicule, I'd do some heavy research. (And hope the boss never finds out. {grin})
Anyway, don't even consider RAID5. That'll be double-dog slow. You need to go mirrored. Yes, that's more expensive. However, I can't imagine a database vendor recommending anything but mirrored. Actually, Oracle says folks should use RAID 0+1 which is mirrored stripes. We don't but our system was setup before that was the recommendation.
In terms of drive size, the more spindles you have the better. That means, buy nine-gig, not 80-gig, drives. Of course, with your sizing requirements, you may buy 18-gig drives instead. However, my advice still stands. More spindles is more speed.
Drive technology will be determined by what vendor you choose. We're an IBM shop and use SSA drives exclusively for our RS/6000 systems. It's fast, allows multiple paths (up to eight) to each drive and allows for easy clustering. Since you will have a large amount of drives, it's more important to have multiple paths to each drive than a single ultra-fast backbone. (Ie: sharing 120 mbps across 40 drives isn't as good as sharing 40mbps across groups of 10 drives.)
As a shareholder of EMC, I highly recommend their products. They are the best bar none. If you are a big player (Wal-Mart, Charles Schwab, etc.), their cost isn't much more than a less qualified solution. (Of course, I don't think you'll be a big enough player to get the really good discounts.)
Overall, the best advice I've seen in this thread is to ask your database vendor what you need to buy. Oracle|Sybase|IBM wants you to have a good database experience and will not give you bad advice on the hardware front. In fact, Oracle (who I most often work with) can sometimes help you to get bigger discount (no one ever pays list price) out of IBM|Sun|DEC.
You're in over your head. Make sure you follow the George W. game plan and get yourself some fine advisors.
InitZero
Having worked in a datacenter with several high end databases, I can't recommend EMC enough. Their disk is fast, reliable, and robust. Their Symmetrix line has onboard cache (gigs of it), direct SCSI connects, fully redundant everything, and the best part of all, EMC support. The arrays dial out when a disk goes bad (or anything else for that matter), the next day an EMC tech is at your datacenter with disk under arm ready to replace it. You really never have to worry about disk problems again. You can even do firmware upgrades with the disk online!
If you are running a medium to large database like a few terabytes. I would recommend investing in their TimeFinder solution, which allows you to make exact copies of the database (or data) by splitting the disks via a third mirror which they call Business Continuity Volumes (BCVs). This makes backups quicker and easier(simply split the third mirror and mount the volumes to your backup host), database schema changes less risky (split the mirror before the change, and you have a speedy backout), and overall your life easier (you will sleep at night).
The above is making the assumption that you are also using a real database such as Oracle that can handle raw devices, and online backups, multiple nodes, etc.
If you can't afford the Cadillac, you can go for the lower end which is their Clariion arrays which is also a damn good little unit if your budget is tight. You can essentially do the same things, except it's not as slick. Either one of them can go up to several terabytes of storage per unit.
---Hey, it's me.
No.
My last SAN project was 5TB raw disk and the solution that EMC pitched me was close to 1Million dollars more expensive than the competition.
In addition to being cheaper, the alternatives were mostly faster than EMC disk *AND* they played nicely with fiber channel hardware from other vendors (unlike EMC which likes to lock you in to their hardware only).
Not to knock EMC (solid product & killer support) but their obscene (no other way to describe it) pricing only makes their stuff worthwhile in situations where you need a 'black box' solution where some other guy is on the hook for hardware failures. In the life sciences I see EMC disk being used on drug manufacturing process hardware as well as on databases and systems that come under FDA scrutiny (patient outcome and clinical trial data, etc.) Generally people with more money than sense purchase EMC for anything but the most absolute mission critical stuff.
The other thing that annoyed me about EMC was the overly agressive frat-boy style sales force. The internal competition to make sales quotas is killer I've heard.
I ended up going with Brocade fiber channel switches (Silkworm II) and Compaq StorageWorks disks. We needed a SAN that could talk to NT, Linux, Tru-64, Solaris, HP-UX and Irix systems all at once.
I'm not a Compaq cheerleader but I like the StorageWorks line because although they are not always the first to market with the latest buzzword technology when they do come out with a product it is generally really solid and actually reasonably priced. The other cool thing about their new universal drive form factor is that all of their disks are now plug and play from the lowest end proliant server all the way up to their high end systems.
As for RAID levels and such you really need your database architects to tell you what they need. It may end up being a mix of RAID 1+0 and RAID5 for some filesystems and they may ask for solid state disks to store indices and such. Hardware tuning for high-end databases is a whole field in itself and there are lots of people out there who can probably tell you exactly what should be needed.
What you are going to find at the end of this project is that disk capacity is pretty simple and easily handled. The real problem you are going to have is figuring out how you are going to backup 5TB worth of DB data :) Not a trivial task by any means...
just my $.02
-chris
The first thing is to talk to your DBA and get his/her input. DBAs, competant ones, have done a lot of this type of work in the past, and they'll have an enormous amount of help to provide you. They'll know your usage pattern by heart, and be able to provide you with some help as to usage.
The first thing to realize is that for most RDBMS usage patterns, RAID is a Very Very Very Bad Thing. But when I say "most", I mean "most with updates to live data."
RDBMS' use data in 4 main types of storage, and it's important to understand them:
- Main Table Storage. This is where your data actually "lives", and is ironically the least important storage wise.
- Temporary Table Storage. This is the storage space for temporary space and temporary tables, which is extremely useful for performance management.
- Index Storage. This is where data indexing structures live, and is extremely performance critical.
- Log Storage. This is where the log for your system lives (physical and logical) and is also extremely important for performance.
The most important thing for performance is to PHYSICALLY segregate ALL four types as much as possible. For example, if you're going into multi-terabyte databases, you might want all four types of data not just on different disk arrays (i.e. RAID controllers), but also on different SCSI channels, and different host adapters (i.e. multiple SCSI or FC-AL cards) altogether.You also want to bear in mind that your update speed is limited by the ability to handle log writes. Log writes aren't limited by bandwidth. They're limited by the latency of each disk. Every disk can handle a certain number of operations per second. Even if you add more disks in a RAID configuration, you're never going to be able to handle more transactions per second, because you're not increasing the number of operations of any of them, and all of them must be touched for every transactional write.
So with that in mind, allow me to recommend something:
The number one advice I can give is to consult with others. If you haven't done this before, there are people (your DBA, your database vendor, your hardware vendor, your systems integrator) who have. This is serious business, and not something to screw around with. Terabyte-level databases are still NOT so common that everyone can and should attempt them. Having terabyte-levels of data throughout an enterprise is, but in one application it isn't. You'll probably not get it right the first time, so take your time and consult with every one of your vendors on capacity and performance planning.
Not to be crass or mean, but if you're asking slashdot, you probably shouldn't be doing this all by yourself.
Good Luck.
The cure of the ills of Democracy is more Democracy.
Erlang Developer and podcaster