Slashdot Mirror
SCSI vs. SATA In a File Server?
turboflux asks: "I'm currently in the process of replacing an aging file server with something more robust. Company-wide, there will be about 100 people who could be using this server, but I don't imagine there being more than 50 concurrent users. Right now, I'm torn between spending alot on SCSI hardware, much like our other servers, or spending less, but getting more space, with SATA II drives. Whatever I decide, the server will be setup with a RAID 1+0 array for the numerous benefits it offers. Does Slashdot have opinions or suggestions on performance, reliability, and stability?"
Have you ever thought about the benefits of RLL?
I don't mean this as a troll, but I was under the impression that USB 1.0 replaced SCSI? Or was that desktop only, and it still has server uses? I mean, I thought USB killed SCSI? Or am I thinking of something different?
SATA is fast and cheap; just make sure you spend a little bit more to get the "nearline" storage drives and not just desktop drives. Put them behind a 3ware 9550 and you'll fly.
Fellowship 9/11
SCSI slaps SATA arround like nobody's bidness. It's not just about bus bandwidth. SATA sure beats IDE, but that isn't really saying much.
Blar.
You know, those are the drives offering logs of models running at 15,000 RPM. The ones with controllers which can often take large amounts of RAM. You know, drives for servers.
Clearly the answer is SCSI - go with what you know, servers are not the best computers to experiment with random equipment.
I'm starting to think GNU is the problem with "GNU/Linux" these days.
Think about the fact that you need a sufficient Backup. You can buy lots of cheap storage with SATA Disks, but Ultrium 3 Tapes (400/800) are still expensive as fuck. Never forget that cost when calculating.
OTOH, there are 300GB U320 Disks now, which you could use if latency is not an issue. Otherwise, go with lots of disk arms (72GB or 36GB U320 Disks)
We have both SCSI raid (2 1TB arrays with 10k RPM SCSI drives on a dell powervault) and a several arrays with 3ware cards (an 8 way and a 12 way both with 200 or 250GB drives). We run Red Hat WS. We find that the 3ware cards are excellent for large data storage but have latency issues compared to the SCSI raid array. We are happy with both systems, but the price break on the 3ware shows, and I wouldn't recommend for really heavy use.
Laboratree - Scientific collaboration based on OpenSocial.
It's more reliable, as far as I know, compared to SATA. SATA is good enough for desktop performance, but I have yet to hear any glowing reviews of it in the server market.
"Better to be vulgar than non-existent" -Bev Henson
More like "harbl harbl harbl!"
I use SATA on our smaller, non-mission-critical servers. For our data backend, I wouldn't touch it with a 10-foot pole.
Here are some scenarios where I wouldn't hesitate to use SATA:
- You have redundant servers. Using LVS and/or Heartbeat and your favorite tools, you can get full server redundancy using less expensive hardware. The overall solution can be quite elegant, with hot failover. Why just cover the drives?
- Front-end cluster nodes. You have a powerful, expensive backend server (with a cheaper failover) and you use inexpensive front-end servers for serving client requests. Sounds like overkill for what you want, but with the right server load balancing technology, it can give you a scalable, fault-tolerant and damn fast solution.
- You can live with downtime. Install a server with a couple of SATA disks in a RAID configuration and hope for the best.
SCSI is great for a database server, but SATA will be much more practicle in a file server. That's the way I would go.
Go SCSI. It's pretty amazing that there are so many people out there whose only exposure to SCSI has been a mention in a textbook or having read about it, considering it was the standard for performance for so long. The speed and reliability is definitely better than SATA and as you well know, price isn't much of an issue when the RAID fails is it?
-- Eli Juicy Jones
For any tier 2 storage, SATA is the future. I'd go so far as to say that in about 80% of all instances, good SATA-II drives with NCQ and large caches in a proper RAID setup will be far, far more than adequate for most production servers unless you do tons of non-sequential I/O, need tons of iops, etc.
For a file server, you'll be fine with SATA.
For my tier 2 servers, I am moving a ton of stuff off of my EMC gear (because fibre channel drives are damned expensive) onto SATA-II drives in an iSCSI setup. I'm already running servers with trunked gigabit NICs... might as well let Win/*NIX boot from a local drive and mount block-level iSCSI devices over the gigabit fabric. Save a shit ton of money and get tons of space on the cheap (7.5 terabytes of fast SATA-II in an iSCSI chassis for well under $10K... make your RAID groups, carve up some LUNs, and par-tay!).
There might be some benefits to you to sticking to SCSI vs. SATA, it really depends on your preference. Both SCSI and SATA offload the main processor from the duties associated with reads and writes. SATA also now has optimized reading patterns just like SCSI. The only real advatages SCSI has right now are the speeds (SATA 150 (there is a newer faster one coming) vs SCSI 320). Also, most SCSI drives are desgined for 24/7 use, whereas most SATA drives are designed for desktop use. Just make sure the SATA drives you buy are made for Enterprise level operation. So it really comes down to compatability/speed vs. cheap/larger. Hope this helps!
Sabre
If you go with SATA 150, make sure that the drives and the controller support NCQ. This is incredibly important for a server. If you want to split the difference between SATA and SCSI, go SATAII with a PCIx 64-bit or PCIe RAID card with expandable on-board cache of around 128MB.
SCSI what?
Why? Why? Why?
At least give us a 2 line explanation, so I don't think you're speaking crap that you read in an advertisement! Please, if you have some justification, I'd love to know what it is... seriously.
Parent is right, for an admin the scariest thing you'll ever here is.
"I can't find the network drive"...
SCSI will seem a bit more expensive at first but that cost isn't just for the interface most of that cost is for the extra testing and hardware reliability you get with SCSI.
I am a pir8 and I back up everything important so I can run an SATA raid-0.
But you want something with modular controllers hotswappable raid arrays and reliability, hell if I was running a businnes off my home PC that would be something I would invest in.
Find a way to make do with less space on the netword and greater reliability, you will sleep A LOT easier.
That thought had crossed my mind as well. I plan to setup the Active Directory equiv of home directories for each user and have them use that rather than My Documents. Plus there will be some common shares with high resolution product images, brochures, and so forth.
the SATA II spec is quite a bit different from the original SATA. SATA II adds port multiplication, hot plugging, native command queuing, external enclosures, and port selection. Also, with a theoretical peak of 3Gbps, it's twice as fast as the old SATA. here is a decent article with more explanation.
Serial-attached SCSI. 15K RPM drives, SAS RAID 1+0. Heaven.
If you like what I've said here, and want to read more, go to http://www.krillrblog.com
You have redundant servers.
Exactly.
SATA is so much cheaper that it's easy to afford completely 2 or 3 redundant storage servers for the same price as a single SCSI array. We do this at work in fact.
With SCSI around $3-5 per gig, and SATA around 35-70 cents per gig, it's a no brainer if you need lots of storage and latency doesn't have to be top of the line.
If it's latency bound then SCSI or more expensive SATA 10,000 RPM Raptors would be better.
I've had enough abrasive sigs. Kittens are cute and fuzzy.
USB 1.0 didn't kill SCSI, Steve Ballmer F**KING KILLED SCSI!
Now Ballmer says he is going to F**KING KILL USB.
Man I need a T-Shirt that says that.
"Steve Ballmer says, "I'm going to f**king kill you!"".
Modesty is one of life's greatest attributes
SCSI is very fast, and usually more expensive. You can get really fast, highly cached drives in SCSI with high-RPM spindles, and cool controllers. But they're $$$$. Do you need the speed?
If not, SATA is still pretty fast, much less expensive, less clever controllers, but still very reasonable for things like archiving, steady low-concurrency-demand streaming, and so on.
SATA also has the advantage of not needing loads of austere cables with distance limitations imposed on them; it's a serial rather than a parallel bus-- hence the S in SATA. Use SATA when you don't need the absolute fastest you can get-- and you won't have to spend the most on the controller (which is hopefully a SCSI PCI-X controller or other fast clocker), the drives, the pricey cables, and so on. But if you need the speed, there is no faster than SCSI except for flash drives, which are still hideously expensive.... and not writeable as much as we'd like them to be.
---- Teach Peace. It's Cheaper Than War.
I recently did all this research myself. SATA on Linux is going to get MUCH faster, probably as fast as SCSI, but you'll have to wait for the libATA improvements to take hold. Right now NCQ isn't implemented, and neither are 'multiple sector transfers'. I bought hardware that WILL support those features because I know that NCQ will dramatically improve speed and latency (under high-use conditions) when it is finally fully-baked.
The site to track progress on the library and driver status is here: http://linux.yyz.us/sata/
The project has been moving along quite well. I think their goal is to completely modularize, simplify, optimize, and consolidate the ATA, ATAPI, and SATA kernel pieces into one overarching (underlying?) library. I like this kind of work. I can't see why ALL disk-like I/O isn't under one big modular kernel library, it seems like it would make adding new transport types and drivers a lot simpler and reduce maintainance all-around.
"Sometimes, I think Trent just needs a cup of hot chocolate and a blankie." -Tori Amos on Nine Inch Nails
Sata has extremely fast sustained throughput... scsi, as I recall can perform more drive intensive tasks while keeping them OFF the cpu mix. When many requests for files come in, either of these will do, but personally I'd put money in SCSI unless you're really a badass geek, have some 3ware hardware, and as in "badass geek" you eat, drink and sleep HDD troubleshooting. (I'm still using parallel drives for mass storage, but they're mirrored on a pair of RAID cards. About 400 gigs, redundant, or almost 1 TB if the drives were separate)... if any of mine fail, its a quick trip to the store for another 160 GB seagate :)
~D
" What luck for rulers that men do not think" - Adolf Hitler
I'd say Fibre Channel.
One benefit that SATA does have over SCSI is the cabling....it's smaller and blocks less airflow (and easier to do the cabling).
SCSI on the other had has other benefits....like it's used in enterprise servers now. Faster, daisy-chained, more RAID options, etc.
Of course, Fibre Channel is basically SCSI on steroids and has the cabling benefits that SATA has.
With more room thanks to less data cabling, u can add watercooling to reduce the heat generated by the 15k+ rpm drives.
I am a huge fan of efficent, cheap systems. The bulk of our server load is handled by dual opteron machines with 3ware cards and a 10k rpm system spindles and 7.2k rpm data spindles. However, even the best sata drives choke under file system and database loads and our primary data stores are U320. StorageReview.com has a good review of the new 150gig 10k rpm WD drive that shows it gettting spanked by SCSI drives under non-linear server loads. Long story short, if you expect a lot of drive activity you might be able to eek by for a while with a well tuned SATA system but you will have to pony up for a SCSI system at some point and you might as well do it now and save yourself the hassle of migrating later.
SAS hardware is currently a little harder to find than SCSI or SATA stuff, but I'm sure there's a good selection out there if you take the time to look.
I was checking out the Sun Fire 4100 a while ago, and it takes SAS drives, however the form factor is 2.5", and I haven't yet seen any 2.5" SATA drives (I wanted that compatibility). Also, I've heard SATA drives don't work with the Sun Fire 4100's SAS controller anyway. Not sure about that, since the SAS spec says they should work, but just something to keep in mind when you're looking for a server or mobo or controller that supports SAS.
One way to curb some of the cost, I might add, would be to switch to something like RAID 5... you won't have as high throughput, but you'll still see performance gains and end up with more usable drive space. The throughput likely won't be your problem, anyways... typically it would be the drive's ability to handle multiple simultaneous requests, which heavily relies on low access times (which is why SCSI dominates in this type of environment).
Here's a quick reference of some IOMeter benchmarks using a file server test pattern. You'll see what I mean. Wealth of info on drives on that site.
You cannot buy the same performance class of drives in SATA as you can with SCSI. Some people call this a market segmentation scheme, I call it catering to the market. People who demand top class drive performance typically also want the other benefits of SCSI as well. Whether those benefits are needed for your requirements, well, depends on your requirements.
SCSI can (depending on which particular SCSI) provide you with more devices per controller without sacrificing (any noticeable) performance. If you need to shove a ton of drives into one server, this will add up quickly. Since you are talking about RAID 0+1, depending on how much storage you are shooting for, this may be a strong factor (but you may be able to skin by on the 4-6 SATA ports you'll find on most mobo's).
SCSI is more mature. So drivers are likely to be more robust, more efficient, and more stable than those you'll find in your garden variety SATA.
You'll typically find that under heavy load, SCSI performs better. Again, this is mostly due to so called "market segmentation" schemes, but that is why you pay more. If your users are going to be mostly dealing with the usual, periodic saving of word processing documents, spreadsheets, and a couple of light media files - you probably don't need to handle really heavy loads. The RAID controller will eat the peaks of write demand in cache (if you get a decent RAID controller - see later), and you should have fairly smooth performance. Then again, if your users are constantly running large installers (development test environment) or working with large remote files - you should really go SCSI.
All that said: I think you would be served best by investing in a better RAID controller rather than investing in top of the line drives. The RAID controllers they integrate on to most motherboards are crap (for what you are trying to do, desktop use - meh). You want something with a ton of cache, and good management soft/firmware. If you buy a real server class motherboard, you may get a better onboard RAID, but however you go about this - pay the most attention to this detail. Unless you really need low latency for high demand, random access applications, top end drives probably won't give you much over the usual network latencies.
I bought into the whole "SATA is the new less-expensive SCSI" and put in two new file servers using SATA last spring. I can say that I'm unimpressed with the SATA servers as compared with our SCSI servers. I now wish we'd spent the extra $1000/server and gone with SCSI. I recommend SCSI -- you won't second guess your decision down the road.
BTW, we used 3ware controllers with WD RAID Edition HDs. We're supporting approximately 75 users per server.
Might not even be SATA II (I'm not sure), but avoid using Seagate Barracuda drives with RAID controllers based on the SIL3112 chipset.
There's an incompatibility with some of the drives, Seagate won't say which ones, and it will result in reduced performance, or the risk of data corruption.
There's a really good page about it, explaining the issue, but I can't find it at the moment. "15m" comes to mind, but that probably doesn't help. Google for references to "seagate errata fix".
As a matter of fact, you may not be flying at all. It all depends what you're using it for. The problem with SATA is latency, and there's not much that controller is going to do about it. If you've got a server that is performing latency sensitive tasks, then SATA can cause performance problems.
In my experience, if you've got alot of random I/O, SATA is not a viable solution. That said, even if your I/O is mostly random, if there's not a heavy load on the disk, then you're probably ok. If you've got 200 people hitting a database or email server, you're probably going to have some performance problems. Swap it out with SCSI drives, or a quality disk array, and you'll be doing much better. If you've got a web server, or a database server that is exclusively reading, you can probably get away with SATA. Again, it all depends on how much and how random the disk I/O for your application is.
Well, SATA 300 is already out. We're using that for 3 1TB file storage servers on our domain, using a PCI-X x16 Raid controller with RAID5.
And to be really honest, SATA 150's 150MB/s is not shared with any drives, where as SCSI 320's 320MB/s is shared among all the drives on the ribbon cable, AFAIK. I'd personally rather have 8 drives with 300MB/s a peice than 8 drives sharing 320MB/s. The former makes the PCI-X port the bottle neck (or perhaps the raid controller card itself).
SATA's peak raw transfer rate (150 MB/sec) is half that of the peak raw transfer rate of SCSI (320 MB/sec), but you're going to be limited by the individual hard drive's transfer rate anyway. Keep in mind that a proper SATA implementation will be 150MB/sec PER DRIVE, since each drive is on its own channel. SCSI is 320 MB/sec per channel, but you're in for a cabling nightmare if you want only one drive per channel. Note that there is a 300 MB/sec SATA standard, although few drives and controllers seem to support it.
If you buy the right model, you can get SATA drives that have gone through the rigorous quality control testing that has historically been reserved for SCSI drives. Many of the higher end server-grade SATA models are warrantied for 24/7 operation. SCSI has lost its advantage there.
SATA has Native Command Queueing, formerly a SCSI-only performance feature. Note that it's optional for SATA drives though, so make sure you get a controller and drives that support NCQ. Again, one of SCSI's few advantages has disappeared.
Last, but most definately not least, SATA cabling is far simpler and robust than SCSI cabling. SCSI cabling is a finicky nightmare where even high-end cables can cause data corruption if you're not careful, whereas even the cheapest SATA cables I've seen worked reliably. I've had hardware related data loss on hard drives twice in my life. One case was an IBM Deathstar, the other was a SCSI cable that started flaking out and corrupted data on three drives at once. I haven't touched SCSI with a ten foot pole since that incident.
retrorocket.o not found, launch anyway?
For some things you NEED SCSI, for others you don't. That much is obvious.
Large files/streams that require heavily mixed-mode I/O beat the balls off of SATA. E.g. Correct me if I'm wrong, but my partial understanding of SATA is that if many writes are cached and a read enters the queue, the cached writes are trashed.
so if you are working with check-in/check-out I/O type such as Samba profiles, SVN stuff, or (Samba|N)FS on a small-medium number of small-medium size files, or web stuff, SATA offers best price/performance ratio, with RAID or whatnot.
If you are working with large files that get a lot of unpredictable I/O, or databases, you really want SCSI.
SCSI drives should be more reliable and come with faster spindle speeds giving low latency and faster random access.
I've heard that some h/w raid controllers use nice fast cache (similar to graphics cards), but, if you design the system correctly, you can get a (much) faster array using s/w raid. On the other hand, some h/w raid controllers have battery backups on their cache and (supposedly) cope ok with an OS crash, which means you can use write-back more reliably.
Spec the CPU highly if you go for s/w RAID, and get lots of RAM. Also, I prefer an aggressively caching filesystem like XFS, but make sure you have a UPS, and don't piss about with s/w that can easily crash the system (like kernel models/drivers/etc).
I've been told that RAID10 is best for random accesses (ie multiple applications/users).
I think my choice would be 15K SCSI drives on multiple controllers with s/w RAID 10. Perhaps, 2 contollers each with 3 drives. RAID1 over 3 drives on each controller, then RAID0 over the 2 controllers. Buy more pairs of drives and you can hot-replace one mirror (a pair of drives) and use it as a backup, just like tapes.
If that gets too pricy, try SATA.
Be warey of benchmarking sequential performance - SATA/EIDE will look good compared to SCSI, but the latter should be (much?) better for random accesses.
Depends on how important reliability is... generally SCSI means better support and warrenties, and hard drives designed to run 24x7. Hard drives that are targeted to the desktop user are designed for desktops, not servers. However if you have a good backup system, and you simply can't afford SCSI, then ya SATA would be your best bet. An example of that is my home server... no WAY I'm forking over a few thousand dollars just to serve my personal files :P
RAID 5 is noticably slower disk performance for writes, and radically slower performance for reads and writes if you lose a disk. In many cases, the performance during a failure for RAID 5 can reduce system capacity below the required level, and thus RAID 5 is simply not an acceptable technology for those environments.
If just sticking more data on disk is the requirement, and you don't care how slow it gets if you have a drive fall over, then RAID 5 is great. But real world enterprise environments exist where losing half your disk throughput will cause the company's service to go down, and then you're out of business. Those guys don't RAID 5 if they know what's good for them.
RAID 5, unless you like 50% waste, rather then 1/n (n>=3) waste.
> Since you are talking about RAID 0+1
Actually, he said RAID1+0, which is not the same thing...in RAID0+1 (with 2 mirrors), if you lose a drive, you're back to RAID0 (no redundancy)...in RAID1+0, you still have a mirror on half your stripe.
The very definition of RAID is "Redundant Array of INEXPENSIVE Disks". Emphasis mine.
I've already read a bunch of posts about how SCSI is more reliable than SATA. Well, they actually mean SCSI drives are generally more reliable than SATA drivers (and some actually say so). They're quite correct for the most part.
Here's what storage vendors don't want you to know: It doesn't matter.
Use RAID. With SCSI or FC disks, you'll have to use RAID5. At that point, two disk failures in a given array and you're screwed. You REALLY care that two disks don't fail at the same time. And when you're using low-end or even mid-range drives, it happens.
Why do you have to use RAID5? Because with SCSI or FC disks, RAID5 is the only economical option. With a 300GB SCSI drive going for at least $1200USD, and FC drives of that size going for $2500USD, even the biggest corporations end up using RAID5.
Of course, RAID5 isn't the only level of RAID. It's the least redundant of any level of RAID, as a matter of fact.
Go SATA with RAID10, at least 4 drives, ideally six or more. With six drives, the likelyhood of having two drives fail before you can replace the first one is somewhat higher than if you're using SCSI, but the likelyhood of that second drive causing you data loss due to a failed array is infinitesimally smaller. It's guaranteed with RAID5, and the chance for RAID10 is inversely proportional to the number of disks in the array. So first the first drive has to fail, then the second drive which fails has to be of the same RAID1 set. Add onto that that drives do indeed "go old", and the heavier you work them, the faster they get old. With RAID5, disks tend to get worked a lot harder (without any cache, or if the cache misses, each write requires n-2 reads, and 2 writes).
Of course, you've pretty much decided that RAID10 is the way to go. At that point it's cost. If you're looking for 50GB of fast redundant storage, SCSI is going to be slightly cheaper. If you need any amount of storage though, SATA is going to be a whole lot cheaper for the same level of reliability (which requires more spindles), and typically better speed (more spindles means more seeks per second and more megs per second, though one needs to be mindful that big SATA disks are only 7200RPM, while the slowest SCSI disks you're going to get are 10kRPM).
Summary? I'm value-concious. I'd go the SATA route. RAID10, four disks minimum to start, a pair of 4-port 3ware SATA cards with 128MB+ of battery-backed cache. I'd do the RAID entirely with software (Linux MD), with each RAID1 set split across two controllers. We get cheap disk redundancy, cheap disk speed, cheap I/Os, and cheap controller redundancy. I'd consider using less fancy controllers, the 3ware jobbies tend to be expensive, but when you're doing big writes the cache makes a massive difference (75MB/s across four disks of RAID10 versus 20MB/s). I've considered putting together a dedicated storage appliance, exporting via SMB/NFS/NBD/GFS/what-have-you, without the battery-backed cache, but with a pair of 1U UPS units (one for each power supply). Then I'd go around turning off all the application-level fsync()ing, and see what happens with 4GB of disk cache. Bet it'd be fast. And with shutdown initiated via UPS trigger, almost as safe as a battery-backed cache. Remember: "Redundant Array of INEXPENSIVE Disks."
God I ramble.
Barclay family motto:
Aut agere aut mori.
(Either action or death.)
I've never understood. . . why the hell would you want hotplugging for internal components? Isn't it always a smart idea to turn your PC off before you reach your hands inside the case?
www.linuxpenguin.net
Yes. All sysadmins should take advice from a slashdot-posting "pir8" that's dumb enough to use RAID-0 on consumer level drives.
Or Serial Attached SCSI.
Higher throughput than standard SCSI, easier to manage and daisy chain and somewhere I'd read that you could attach SATA drives to SAS controllers - although that's never been confirmed.
Who is general failure, and why is he reading my hard drive?
It's not the interface itself. Interface incompatibility is used to split the market into regular users and those who don't have to spend their own personal money.
Think about rotation speeds, seek times, and bearing wear. How could these specs possibly have anything to do with the data cable? They don't, except in the mind of a VP of marketing. Despite the obvious lack of technical reasons, SCSI drives often come with better specs. That's life.
Use SATA and SCSI.
There are devices available that appear to the comuter to be a SCSI drive when it is really a RAID array of SATA drives.
Something like the the Maxtronic Arena Sivy SA-4830/SA-4831 could give you a 2 TB SCSI drive.
All hardware (and software) sucks, and it breaks, it's a fact of life. No matter if you go with SCSI or SATA, the important thing is that you can find out when a drive dies so that it can get replaced.
Many low to mid range SCSI raid cards (most? all?) either don't have any sort of interface to find the raid status when the server is up (they just beep at you and expect that somehow that's going to be hard over the AC and server noises when you're walking by the machine), or the tools for checking the raid status are so poor that they'll lock up the shared memory segment after checking a certain amount of times (ADAPTEC, I'M TALKING TO YOU). Since being certain about your raid status means checking it via something like nagios, that means that it gets checked many times, and will thus eventually lock up.
While SATA is nowhere near the performance of scsi (despite what SATA fanboys will tell you), 3ware cards are actually really good at:
a) letting you know when a drive has failed
b) letting you check with their tools as many times as you want without locking it up
And since the SATA stuff is so much cheaper, you can buy multiple servers, so even if the card fails, you have a hot backup.
If you absolutely have to have the fastest, go for a raid 10 of 15krpm drives.
If you don't, and want peace of mind, get at least 2 SATA setups with 3ware cards.
SCSI is obsolete and being actively replaced with SAS (Serial Attached SCSI). So the question should be whether SAS or SATA should be used instead, that would be a step in the right direction.
Also note that SATA drives may be used on SAS controllers. So the answer to your question could ultimately be something like: a mix of SAS and SATA.
The one thing that is really different and might be substantial depending on your application is that SCSI has been shoe horned in to more exotic configurations and because of that there are more standards for doing "enterprise" type things. Like running a heart beat over SCSI, or have a SCSI array attached to redundant hosts. I wouldn't kid myself in to thinking you could do anything of the sort with SATA in a remotely standard or supported way. Some of the big storage players are getting in to SATA though.
Performance between best SCSI and best SATA is nearly identical. SCSI is more configurable. SATA gives you more bits per dollar. SATA is probably more "leading edge" where as SCSI is old technology that is tried and tested. Depending on the applications, maybe what you really want is a storage device and a server with whatever the boot drives it comes with and a dedicated high speed interconnect. If you're really serious about storage, quit fucking around with SATA and SCSI type questions and really solve the problem.
Yes, that's why many big NOCS end up mirroing raid5s and other fancy things. The level of performance you get is directly related to how much you're willing to spend per gig of storage.
Are you willing to spend for 10 terabytes, just to get 1 terabyte of usable space? Well, then don't expect to have incredible throughput while experiencing the most catastrophic failure imaginable...
You'd be crazy to do sata raid without a dedicated controller card. In that case, ide/sata has the same CPU utilization advantage scsi.
Serial Attached SCSI takes the best of both world together:
SAS has:
- lean SATA cables
- 3Gbps transfer, soon to be 6Gpbs. Better than U320
- 15,000 rpm disks
- NCQ like SATAII
- RAID-capable controllers
- SATA on SAS possible
You can buy drive enclosures that allow you to hot swap hard drives.
So NCQ or whatever doesn't cut it?
I guess I can understand what you mean though. It should also be clarified that it's not just any SCSI drive that you need, that it would need to be an array of 15k RPM drives if latency is the real issue because there do exist 10k RPM SATA drives with decent capacity that is more or less the same drive mechanicals with a different interface as a SCSI counterpart.
LOL these kind of post always make me laugh to tears.
OMFGROTFLMAOLOL!
50 concurrent users is a LOT. You may not really mean concurrent, as in "50 people actually reading from or writing to this drive at the same time". If you DO mean that, you desperately need SCSI, the fastest you can find. You'll need seek time more than anything else; the drives need to respond as fast as possible to multiplexed requests for data. Rotation speed, which improves seek time and transfer rate, is good too, but it's seek time that's most crucial in heavy multitasking environments. If by 'concurrent' you mean '50 people occasionally hitting the disk', then yeah, you could probably do SATA.
However, you already have SCSI. Management is used to paying for SCSI machines. If you have 50-100 people depending on something, and it's slow, that's a productivity drag. If you assume that all those people cost $100k/year each (not at all unreasonable with benefits), 50 people are getting paid about 2,500 bucks an hour, or about 20,000 dollars a day. In other words, if you speed them up by just 5% with better hardware, you're saving the company a thousand dollars a day. Even if it's a tiny 1% speed gain, that's still 200 bucks a day. Saving six grand a month for an upfront investment of ten grand is a total no brainer.
Buy SCSI.
.. if you do end up with SATA, make sure to get some neon lighting for inside the case.
--
Current setup - 4x Seagate 400GB SATA, NVRAID-0, ThermalTechno 4000 1U Case w/ Ground-FX, 3x Zalmat 80mm SilentKiller Fans (soon)
If my limited experience has taught me anything about computer reliability, it's that a single mis-set bit somewhere can bring down a system. Maybe the bit got there by user error, maybe it got there because of RAM or disk failure, maybe it got there from a bug in the application, OS, or firmware. Maybe a component on the motherboard shorted out. Maybe it's the climate. Maybe it's the phase of the moon.
I've seen it happen with discount ghetto hardware, I've seen it happen with high end hardware. I've seen it happen on Windows. On Linux. On FreeBSD. On Solaris. I've seen servers go down due to catastrophic hardware failure and I've seen them go down because a $2 fan died. I've seen people come inches from major power supply caused injury working on a desktop PC.
Everything will break.
There's just too much freaking complexity. Now I just buy whatever's cheapest so I can buy way more than I need. Mix up the configurations a bit so you get some bio-diversity; if one drive manufacturer has a bad year, you don't want all of your eggs invested in them.
Most important of all, at the first sign of trouble, throw it away.
Try to resist the urge to fix it. I mean it. You cost more than that piece of junk. Put in a purchase request and move on.
Where are the 15k SATA drives? But other than that, SATA RAID card is pretty good if you just want one giant volume for storage and maybe a disk or two more for OS and swap.
Blar.
It's not necessarily for "internal components". It's also for entry-level servers and raid arrays. You can get hotplug bays that fit in 5 1/4" slots on a machine. This provides an easy way to swap out the drives in case of failure. If you're running a server you nor your users want downtime. If you're running RAID 1, RAID 5 or RAID 10, you want to be able to rebuild to the replacement drive before you lose another drive and lose the whole array. Its a lot faster to rebuild a single drive from the existing data on the other drives, than having to restore from tape or other backup media.
Also motherboard manufacturers are now starting to include external SATAII ports to "hotplug" external SATA drives.
Unless price is that big of an issue. It's been my experience that SCSI will also give you better I/O performance (in part because it can read and write at the same time and that it can have rpms up to 15k).
SATA would be a second choice only if I felt like worrying about when the first disk in the array was going to fail and (hopefully) be replaced.
The short of it, even in hard drives, is: you get what you pay for.
SATA drives over 80-100GB will burn incredibly fast. Relying on those drives for mission-critical data is not a good idea, even if you have a top-notch RAID controller.
Go with SCSI.
lucm, indeed.
Wrong. 1+0 or 0+1 is the most inefficient way you can to store your data. You are doubling your hardware storage when doing something smarter will solve your problem.
You are completely wrong when you say that:
>In many cases, the performance during
>a failure for RAID 5 can reduce system
>capacity below the required level, and
>thus RAID 5 is simply not an acceptable
>technology for those environments.
I beg of you to show me an environment where performance is a necessity, and they use RAID 0+1. IT IS NOT USED IN PRODUCTION. The only place where 0+1 or 1+0 is used is at the consumer level.
When you are dealing with a single file server, such as the original poster's environment, you use RAID 5 because performance hit will be negligible. When a drive fails you take down the server and go to CompUSA and buy one immediately before you lose your pr0n and illegal mp3s.
When you work for a bank, you have racks and racks of hard drives, and they heads they are connected to use one of RAID 4, 5 or 6, or Netapp's vastly superior RAID DP. These filers are specially designed to be used by thousands of end-users, and they all use SCSI for the highest performing models. If they want to back data to tapeless backup, they will use ATA drives, but if this was really an environment where performance is required above a certain level as you suggested, then they would use SCSI.
Unlike you, I actually have experience , so I know what I'm talking about and more importantly, what technology is suited for which type of environment. I don't talk out my ass like you do, giving bullshit statements, dropping misnomers like performance when it is obvious that RAID 5's performance characteristics won't make a bit of a difference to someone who is asking about the difference between SCSI and SATA for a home file server.
Like I said, the difference will be negligible. The performance impact from either the network or from having a fragmented hard drive will vastly outweigh any perceived performance hits from using RAID 5.
Most people don't realize that SATA's NCQ is NOT the same as SCSI's command queueing. SCSI's command queueing has two parts, the command queueing itself, and the disconnect/reconnect. SATA has a spec for disconnect/reconnect, but I'm not aware of any drives and controllers that implement it yet. That's why all the benchmarks of SATA drives supporting NCQ show much lower performance versus otherwise identically spec'ed SCSI drives on fileserver loads. The NCQ drives are much better than SATA/PATA drives without however on fileserver loads.
This may not matter for you if you aren't pushing the limits of performance or cost is important enough, but don't make the mistake of thinking that a drive and controller supporting NCQ mean you get SCSI performance for IDE prices, because you most assuredly do not. Wait a year for drives and controllers supporting disconnect/reconnect (probably not standardized until 600 Mbit SATA comes out late next year) Then you really will get SCSI performance at SATA prices (other than being able to get 15K drives for SCSI and not SATA, but that may change by then)
So, while adding more drives makes the chance that a second failure will hose data less likely, it makes the chance of such a failure occurring more likely. If you have 4 drives, you have two mirrors. Take the probability of any single mirror having both drives fail and you can multiply that by the number of mirrors you are striping to get the probability of data loss occuring. Having a second drive fail in a different mirror merely means that you came 1/2 way to failure twice.
Some SATA drives are approaching SCSI's reliability, but you really don't know what you're getting most of the time. I just two Maxtor drives fail in a RAID-1 array last year in a Dell computer I had. Maybe one had failed and I just didn't get informed, maybe they both blinked out around the same time, I don't know. I decent SCSI setup will have tools to alert you when there's a problem. I think most SATA drives are around 600,000 hours MTBF, but I think that is measured as part-time usage (i.e. drive on for 12 hours, drive off for 12 hours) where SCSI's 1,200,000 hours MTBF is measured as 24x7 operation. If SCSI drives are twice as reliable and you get notifications of problems, then the chance of having two drives fail before you can get one replaced is much smaller with SCI, even in RAID 5.
So many replies about performance and all the great new features in SATA IO... Too bad no one has mentioned the real issue for high end use... All modern scsi drives allow the HBA to make sure data has actually been written to disk. SATA does not. If you lose power, you lose the data in cache. Worse even in a raid - there the data might be written to two of the data disks but your redundancy disk(s) have not. Now, if a drive fails, you'll restore from parity and boom - you have wrong data that according to all drives, controller and filesystem status is good. That's a real bad thing(tm).
SCSI controllers with battery buffered cache will take care of that - Unless all drives have written the data to disk, the line is not released in the controller cache and when power comes back on, it can be written to the drive. That will not keep you from having to check consistency but it will prevent you from seeing corrupt data as good.
Peter.
If you build a RAID from X SATA drives and Y SCSI drives both at the same cost points, you'll get better fault tolerance and performance from the SATA set, simply due to the low cost of the drives. Although the individual drives are less reliable, as a set they are more reliable thanks to the redundancy.
You just need to set it up so if several drives fail at once you won't lose data or hurt performance (RAID5 is notorious for this).
Related question... anyone using DP Raid in anything Linuxy? We're using NetApps all over the place where I am, and DP Raid is pretty nifty. I'd like to get out from under the NetApp burden though
I've been looking at a mac xraid myself. in 3U (plus 1 more U for the server) it's got 14 hot swap bays each running ULTRA ATA 100 with a separate controller on every disk. redundant logic, hot swap power supplies, hot swpa disks, and hardware raid 0,1,3,5. dual fiber channel to the server.
even if you stuff it with apple disks the whole thing including a high performance dual gig dual processor xserve is about 15K$ for 7 terrabytes running at 7200 rpms. You can get it slightly cheaper if you populae it yourself and throw together your own server. but it hardly seems worth it since a major attraction is that you can also get a 3 year apple warantee on it from a reliable company.
So if your network is going to be distributed over ethernet then I can't see why this is not one of the best things going.
I'd like to hear why sata or scsi linux is a better solution.
Some drink at the fountain of knowledge. Others just gargle.
SCSI still tears the alternatives to shreds for price/performance at the heavy end of the load curve, no doubt about it.
If you doubt it, try both.
For going on twenty years it's been the same: those who haven't tried SCSI claim that there's no or little difference. Those who have used both SCSI and [MFM,RLL,IDE,ATA,SATA] in high-load environments hate to try to make due with anything but SCSI.
For performance and reliability reasons both, you want SCSI if you're dealing with high-random-access-load or high-throughput situations. ATA/SATA is fine if you're just offering up noncritical bulk network storage but for the rest you want the real deal, and you will notice the obvious difference if you try both in a stressed environment.
STOP . AMERICA . NOW
Check out the Areca 1160 or 1260 controller card; it's also sold under the Tekram brand. I supports RAID6 (as well as the standards 0, 1, and 5), SATA II (or SATA 3.0Gb/s; whatever), and NCQ. As far as drives go, I'm very happy with the Hitachi 7K500, just make sure you turn on SATA II support and other advanced features that they ship turned off. It has good support under Linux and Windows (probably OS X, too), and is real hardware RAID. It supports a BBM (72 Hours, IIRC) that will allow you to turn on writeback caching for extra performance. The cache is a standard laptop SODIMM; it ships with 256M cache and expandable to at least 1G; I believe it will support a 2G SODIMM if you can find it.
I'm running 1 TB (RAID 5) of data in this setup right now, soon to expand to 2.5 TB (RAID 5). At some point I'll upgrade to RAID 6, but the card supports 16 drives, so that will still give me 7TB of data. If you need more data than that, I believe Areca also sells versions of the controller that support 23 or 32 drives.
(If you wanna do SCSI, Areca has controller cards for that, too.)
- You get extra read bandwidth. Either drive can service the load, while in RAID 5, the parity is pretty useless.
- Degraded-mode performance isn't very degraded.
- You can split the mirror for special occasions. In particular, our emergency evacuation plan involves grabbing half of the main file server's drives on the way out the door. Yes, there's a risk of a failure at just the wrong time, but having an up-to-the-minute backup and not having to shoot the server in the head is very nice.
- No partial-stripe write problems.
The downside is, it takes more disks for a given capacity, which is a problem when you get to the 5-10 TB range.But it emphatically is used in serious applications.
I've used tapes, disks, CDs, DVDs, and even in a few cases printouts of really important data. And tapes are still my favorite.
If the "76" in your username is indicative of your DOB, then I suspect this is a generational difference. I can actually agree with both of you, but having lived through exactly what the parent was talking about due to the problems of tape evolution in the DOS era, I tend to think you haven't.
Personally, I'm a huge fan of using drives, with weekly or monthly tape archives for the longer haul, depending on the needs of the environment. The number of times I've had to pull a 5 year old record versus the number of times I had to get a quick restore from the previous day or week, is extremely small.
The only real downside to drives would be the greater risk of virus infection or intrusion, which is why I prefer to keep weekly tapes. Most of the time I find myself doing restores, is because some user did something stupid, not because of data integrity compromise. I have had to do it a couple of times due to virus outbreak, but it never was able to hit my backup drives. Only use mirrored or striped Raids, mount and unmount them as you need them, file systems encrypted, keep data in tgz format, etc...
No, choose RAID 5 instead of RAID 1+0. Here is why:
To give you a datapoint, I have set up multiple Linux software RAID 5 arrays on various servers with 10+ SATA disks, and the I/O throughput is over 500+ MB/s (enough to saturate 2 full-duplex GigE links !). At my previous work we had about 200 servers, all using Linux software RAID 5. And we have been MUCH MORE HAPPY than the previous setup where all of them were using hardware RAID 5. Moreover, Linux's software RAID 5 is more flexible (create arrays on ANY disk on ANY SCSI/SATA card in the system), more consistant (one and only one control software to learn: mdadm(8), no need to use crappy vendor tools or reboot into vendor BIOSes), cheaper (no hardware to buy), more reliable (no hardware card = 1 less hw component that can fail), easier to troubleshoot (plug the disks on ANY linux server and it works, no reliance on any particular hw card) and more scalable (spread the load across multiple disk controllers, multiple PCI-X/PCIe busses, or even multiple SAN devices).
It's amazing the amount of misinformation and misconceptions about RAID that is spread around the world. I hate to say it but 95% of IT engineers don't make good choices regarding RAID servers because of all those misconceptions.
I would avoid a one-sized fits all solution to this.
Use SCSI RAID (1+0), 15k drives, for your frequently accessed stuff. A second pool of less used, but still accessed drives, say 10k, still in RAID (1+0) for some less used stuff, and then SATA for hot backups, and *really* old stuff that no one ever uses, but you still need around.
Im glad
I am currently piloting a setup intended to replace 35 TB of disk+tape used for sesmological data:
Each array will consist of a pair of full (42-disk) nexsan SATABeast 4U boxes. These use RAID5 + hot spares and connect to the host systems with FC.
The two pairs will be mirrored (over a fiber connection) between two geographically separated locations.
The payback time for this setup is so short that we can plan on replacing half the gear every year, using a staggered schedule. (Replace A after year 1, B after two years, then A/B/A/B etc.)
Terje
"almost all programming can be viewed as an exercise in caching"
If you want speed speed speed, have many users, want hardware that lasts long and have multiple requests to your file server, and can afford it: Go SCSI If you want cheap speed, hardware that wont last long, and a small number of users and requests from the file server: Go SATA The biggest difference being cost. In other words if you can afford it go SCSI. If you can afford it then you'll finally understand the benefits of SCSI, because you'll see it with your own two eyes. If you can't afford it then well..... you just answred your own question.
You're focussing on 1 server with a lot of horsepower. But that single server's going to serve 50 people at any given time. That will make the mainboard/network channels the bottleneck, if not the HDD controller's ability to reach main-memory.
Why not go for 2 servers, use cheap SATA RAID1+0 disks? Then you can move the load to multiple servers and if more demand is required, you're not facing bottlenecks, just add another box.
Never underestimate the relief of true separation of Religion and State.
If you want reliability for the disk you had better check what the manufacturer claims for the MTBF (mean time between failure).
Many SATA drivers have a MTBF of around 0.6 to 1 where as SCSI have between 1 and 2. Your SCSI disk therefore has about twice the life expectancy. If you couple this with the speed of the SCSI I guess for the moment if your budget allows for it then go for SCSI
If your budget doesn't allow for it...just make sure you have good redundancy in your RAID with at least 2 redundant disks
perl -e 'print $i=pack(c5, (41*2), sqrt(7056), (unpack(c,H)-2), oct(115), 10);'
ZFS offers a more secure storage medium than RAID, no matter if you use SATA or SCSI. Here's an article on RAID-Z and the advantages of this system.
There are a number of performance and reliability differences between SCSI/FC and SATA drives.
SCSI (or FC) drives usually spin at 10k-15k RPM, vs. a SATA drive's rather leisurely 7.2k RPM.
This gives the SCSI drives a significant advantage in random access performance (measured in IOPS).
However, the two technologies aren't that different for sequential performance (measured in MB/sec).
Therefore, SCSI drives excell for high intensity workloads such as database servers, while SATA drives suffice for tasks such as large file repositories or reference data.
Another difference is in the reliability (MTBF) and unrecoverable bit error (BER) rate of the SCSI and SATA.
SCSI drives generally have an MTBF of about 1.5M hours, vs. a typical SATA drive with 1.0M hours.
And the BER of SCSI drives is usually 1^15, vs. 10^14 for SATA drives. That's a lot different.
From this you can calculate the mean time to data loss (MTTDL) for a given RAID array configuration.
This is OK for smallish RAID arrays, but SATA gets scary for larger arrays (which start to require dual parity configurations). Because of their higher reliability SCSI arrays don't normally have this concern.
IIRC, the SIL3112 chipset is on the low end of the controller spectrum. Unless someone slipped him a fifty and said "make it happen", he might want something a bit more upscale.
The problem with SATA is latency
Huh?
I'm more than a little disappointed that this got a "5: Informative". Please. There is little to no difference between the SATA protocol and SCSI protocol in terms of latency. To the extent there is, one could easily argue it favors SATA.
SATA *drives* tend to have lower rotational speeds, which tends to lead to slower responses to requests, but you can get SATA drives with zippy response rates that match or exceed the numbers for most SCSI drives.
The SCSI protocol is traditionally preferred for heavy loads and random access patterns not because the low latency it provides, but rather the command queuing (which technically can actually increase the latency of a given request) which allows the disk to minimize how much it jumps around under such circumstances. Of course, unlike traditional IDE devices, the SATA protocol supports this kind of functionality, and it is supported by a lot of SATA devices.
sigs are a waste of space
Assuming equal storage sizes, SCSI drives would have way better throughput and latency than a SATA drive because you can get 15K SCSIs. However, the sizes are NOT equal. Fact is that for the price of a 147GB 15K SCSI drive, you can get about 2TB of 7200RPM SATA space.
What you end up with is the following throughput when disks are empty:
1x147GB 15K SCSI -- 150MB/s
8x250GB 7200 SATA -- 275MB/s to 550MB/s depending on exact RAID configuration
Now fill up both configurations with 140GB of data and the throughput of the 15K SCSI has dropped in half to 75MB/s because the heads are now positioned at the "slower" inner portion of the disk. Meanwhile, the 2TB SATA config is 7%-15% slower depending on the RAID config.
Latency also benefits from many disks for the same reason. Fill up a disk and you possibly have to traverse the entire disk. So while a 15K drive has a seek time of 2-3 times faster, you end up having to move 10X-15X farther than in a mega array where the heads pretty much just hover over the 2X faster outer portion.
The big advantage for SCSI is the better TCQ algorithms for multi-user access. This can be mostly negated if you use a SATA RAID controller with enough onboard RAM to reorder IO at the controller level versus depending on the drive's NCQ.
This is the route we've taken -- we went from a LSI MegaRAID 320-1 + 4-drive SCSI RAID config to an Areca 1170 + 1GB RAM + 24-drive SATA RAID. Every aspect of performance is up by big amounts -- throughput, latency, multi-user access. The drive array is actually TOO fast for our 2x244 Opteron server to drive. We ended breaking the array into 3 8-drive volumes and mirroring 2 volumes against each other for more redundancy. One of these days, we'll upgrade to faster CPUs and retest a 16-drive volume.
There's a thorough comparison of SATA vs SCSI drives at http://www.storagereview.com/articles/200601/WD150 0ADFD_6.html. For multiuser use 10000 RPM SCSI (or its close relative, Fibre Channel), typically performs 150 - 200% more I/Os per second than the WD Raptor 10000 RPM SATA drive. SCSI drives also come in 15000 RPM flavors, which widens the performance gap even more.
I didn't say that sata raid could achieve the same speed as scsi, only that it can achieve the same low cpu utilization.
"Go SATA with RAID10, at least 4 drives, ideally six or more. With six drives, the likelyhood of having two drives fail before you can replace the first one is somewhat higher than if you're using SCSI, but the likelyhood of that second drive causing you data loss due to a failed array is infinitesimally smaller. It's guaranteed with RAID5, and the chance for RAID10 is inversely proportional to the number of disks in the array. So first the first drive has to fail, then the second drive which fails has to be of the same RAID1 set."
p erspective, but why would you wait so long if you can fix it earlier? So it's more likely a situation where you have to look at how large the risk are that one or more drives in a specific amount of time. And sure with 4 disks it's less likely that 2 dead disks means a broken array than with 2 disks, but the chance that any disk dies are 2 times larger.)
:)
The totalt amount of drives doesn't effect the chances that two drives within the same array will die. Sure it's less likely that both drives in the same array has died when only two drives have died in total, but also since you have more drives there will be a higher chance that they do die. If it's 1/2 chance that one drive dies and 1/4 that both dies at the same time that will still be true even if you have 1000 similair arrays.
(You seem to look at it in a what-if-i-wait-so-long-that-two-drives-might-die-
Whatever
But maybe you mean the same data are also stored on all arrays?
0+1 is a mistake, but 1+0 isn't. 0+1 loses the data set if any one arbitrary drive fails in both sides of the mirror; 1+0 only if you lose both disks in a single mirror pair.
Shouldn't that be "01" and "10"? Wasn't the "m+n" notation used for generic RAIDs á la "m drives total, n can fail before the array dies"? In that case a 1+0 setup can even run without a RAID controller - and a 0+1 one is kind of hard to implement...
USE HOT GRITS WITH STATUE OF NATALIE PORTMAN (NAKED AND PETRIFIED)
From own painful personal experience, do not use a motherboard RAID controller for anything other than simple mirroring. Motherboard RAID controlers aren't even consumer grade, and that goes double for Soyo.
"Is life so dear, or peace so sweet, as to be purchased at the price of chains and slavery?" - Patrick Henry
I'm surprised nobody mentioned the obvious. I mean, OK, if you want to limit yourself to drives directly connected to the server, then SCSI wins hands down if for no other reason than you can have 15 of them on a (high bandwidth) cable. Then there's the fact than no SATA drive is designed for enterprise operation (24x7).
But take down the server and all the drives are inaccessible during the downtime.
A smarter solution IMO is a SAN. The whole concept of a "server" is kind of outdated now with SAN (or even NAS). Plus expandibility isn't as big an issue as it is with a "server".
So why in this day and age do people even mess with "servers" for simple data storage? Or did I miss something?
Whatever you do, be sure to read the following article first. It's a bit dated, but it's written by people who definitely know what they are talking about (Seagate engineers) and may help put things in perspective. The hefty pricetag of SCSI does not depend on the interface. Read this and you'll now why.
r /D2c_More_than_Interface_ATA_vs_SCSI_042003.pdf
http://www.seagate.com/content/docs/pdf/whitepape
Being well balanced is overrated. -- John Carmack
I have both. They both work, but I've found SCSI to be more tempremental. You have to get everything just so, or it will have you tearing your hair out. SATA is simple plug-and-play, and a tad more forgiving.
Assuming 4 drives and a smart RAID10 controller, reads can actually be faster than RAID5.
This is because you can actually interleave reads from mirrored drives to get better read performance for single sequential reads. 3ware has controllers that do that. In fact it should be better than stripes, because the same data is present on all mirrored drives so you can read the first chunk from whichever drive with its head closest to the chunk and schedule the subsequent chunks from the rest of the drives, thus minimizing latency and still maintaining throughput. Whereas with stripes, the first chunk will only be on one drive, so you have no choice of drives to read from.
So for sequential access you'd get the combined read throughput of four drives, and the write throughput of two drives.
Whereas with RAID5, you only get the read throughput of N-1 drives and the write throughput is poor (especially for sustained writes).
Of course if your raid controller isn't smart or you are using Linux software raid then for RAID10, you'd get single sequential read throughput of N/2.
Still, Linux software RAID is smart enough to balance reads in mirrors. Say you have a mirror with two drives A and B, and two sequential reads going on, one read typically ends up on A and the other on B. So in many multiuser environments the lack of read interleaving might not be such a big deal. I haven't done tests for RAID10 yet though.
I agree that RAID controllers fail way too often.
You'll need seek time more than anything else; the drives need to respond as fast as possible to multiplexed requests for data.
I have a site that uses 15K RPM U320 SCSI drives in a SAN. I also have a very nice high speed streaming tape library to back it all up. The backups were running around two hours and I felt that, with my disk and tape bandwidth, the backups should have been much faster. Looking for the problem it finally occurred to me that there were over one million files being backed up. Doing the math, I determined that with a 4ms seek time I was spending at least one hour or 50% of my backup time just seeking. Disk head seeks were consuming one hour of backup time on this SAN! I switched to image level backups, where the disks are copied bit by bit, rather than file level backups. This eliminated the more than 1,000,000 seeks (assuming little or no fragmentation) and in turn reduced the backup time to just under one hour.
The point is that most people don't think about seek time and in many cases it is not an issue but, if you have a lot of files or severe fragmentation, disk seeks can be a significant factor.
Of course the same SAN using SATA drives would increase the under one hour backup time to at least four hours but, who's counting.
Did you buy desktop-targeted SATA drives (like the Maxtor DiamondMax series, known for having major problems), or server-targeted ones like the Maxtor MAXLine nearline drives, WD Caviar RE2 nearline drives, or the WD Raptors?
SATA itself blows SCSI out of the water in terms of cabling reliability. It's even in terms of performance features such as NCQ. You just have to be careful not to buy a desktop grade drive. (It's impossible to buy a desktop grade SCSI drive. It IS possible to buy a server grade SATA drive that has been tested as rigorously as its SCSI counterpart, you just have to be more careful with the drive purchase.)
retrorocket.o not found, launch anyway?
I was totally serious. what is wrong with my post?
Some drink at the fountain of knowledge. Others just gargle.
Regardless of whether you use SCSI or SATA or EIDE or FC, you need to know and trust your RAID controller as if your job depended in it. Drives will fail. Some drives will fail more than others. As long as your RAID controller can recover quickly on failed drives, not lose any data, and can be monitored from your OS so that you can detect when a drive fails so that you can replace your hot-spare ASAP, you'll be fine.
Consider how long it will take to rebuild a 400GB drive as part of a RAID5 array compared to a faster-spinning 73GB drive part of a RAID1+0 array. While your controller is rebuilding, your array is at risk for data loss if another drive fails.
If you cannot monitor your RAID controller form the operating system, you ma never know that a drive failed and rebuilt using the spare. You could go days or weeks without knowing that you have no spare available for the next failure.
A RAID controller without a cache battery is slower than one with a cache battery. If your RAID controller safely buffers writes, your operating system won't have to wait for the slow disks to write the data. Turn off write buffering on the hard drives. The write buffer on a hard drive is an area where data can get lost on a power failure that neither your operating system nor RAID controller can recover from.
Keep your disks cool and power-conditioned. Don't put your file server in an 80-degree closet. Don't just stick the drives in a cheap case that doesn't have proper airflow over the drives. If the data is important to you, you will make sure your file server is on a UPS protected from brown-outs, power spikes, idiot employees or even cleaning staff. If you can monitor your UPS from the file server, so much the better.
All of this is more important than SATA-vs-SCSI and applies to both.
When it comes to SATA vs SCSI, it is a matter of low-cost capacity versus performance. If you need high-RPM drives to support low latency times for random read access, you're stuck with SCSI. If just having enough capacity for your data-hungry staff is all you need, then SATA might be a better fit for your budget.
...true multithreaded i/o like SCSI supports natively. Yes, NCQ approximates SCSI-esque multithreaded i/o but under truely heavy loads of random i/o, a raid10 array made of SATA-II drives will definitely hit the performance brick wall long before a similar array made of u320 scsi drives on a good hardware caching craid ontroller card like those made by LSI, Intel, IBM or HP for their big wintel servers.
What kind of file server?
Are you going to be running any database applications on it?
Is seek time going to be important?
If you are just going to store a lot of documents, applications, and static data then SATA is going to be a good solution.
If you are going to do a lot transactions then SCSI is probably worth the money.
My office is about half the size of yours and our file sever has worked just fine using ATA drives for years.
For our next server I am thinking about using a mini-itx system with SATA RAID. We are having space and heat issues at our current office. Now our database server is a different matter.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
Fyi: With Linux the SMART function of the SATA drives will be broken. It isn't supported with the open source drivers and tools and that's it.
CTU had certian information that SCSI was working with the terrorists and Jack got to SCSI first...
I have worked for major system integrations, system vendors, and VARs, and I have architected and installed and benchmarked major storage systems on and off for 10 years. I've installed more full-rack-plus sized equipment than I can conveniently count, including work with Hitachi 9900 series and smaller stuff, EMC, Sun, Storagetek, IBM, as well as NAS gear from NetApp and some exposure to BlueArc.
When I say RAID 1+0 is in use in production, I mean it, because I've architected and built and operated it. At major ecommerce websites. At banks and financial institutions. At retail companies. At ASPs. At biotech companies.
Up until a couple of years ago, large customers used either A) direct connect fiberchannel drives, with FC RAID controllers, and large SAN environments, or B) large NAS systems with a wide variety of RAID configurations. NAS was cheaper but lower performance. For highest performing models you buy Fiber Channel, period. Claiming that SCSI was what the highest performing models use indicates you've never worked with true high end enterprise equipment.Now, SATA drives connected to FC raid controllers in SANs are where the action's at, but all the major vendors still sell FC drives and SCSI drives for applications where the command tag queuing matters (though SATA 3 will eventually make that common across the technologies).
I really don't think someone who didn't even think of FC systems or SANs is in any position to be lecturing me about RAID performance.I know how RAID 5 and RAID 10 perform; I've been working with them continuously since the hardware vendor I was working for started building RAID boxes based on the Mylex DAC-960 controller back in 1995-6 and I got to both write performance benchmark code and use standards like SPEC's LADDIS SFS93. I've talked code with the RAID controller vendors. I've been building RAID systems since 4 gig SCSI drives were brand new, and I use both SANs and SAN attached SATA storage in bulk today.
There are people more expert than I am in this stuff; I haven't built a large storage system using the highest end stuff in a couple of years, and there are clearly people at integrators/VARS/vendors who live and breathe storage every day and are tip-top current experts. But I know what I'm talking about.
And, clearly, you don't.
Hmmm... let's look at an Hitachi USP. This is their high-end storage unit right now and comparable to anything you can buy from EMC or IBM. They support RAID 5 in 3+1 and 7+1 configurations. They also support RAID 1 in 2+2 fashion (haven't seen a 4+4 setup). I have personally consulted at at least 8 of the Fortune 100 and setup RAID 1 for high i/o environments. RAID 5 is generally used for lower demand hosts. They also use fibre channel drives although SATA may be an option sooner or later since you can get them on their modular storage now.
I can tell you I DO HAVE EXPERIENCE and you are full of shit. You're also a bit combative. Have a drink. Relax. It's slashdot.
_damnit_
It's my job to freeze you. -- Logan's Run
I just built and tested RAID1 arrays on the Ultra320 SCSI bus of a couple Dell 1750s.
More than hotswap even, I was able to hot-kill one drive, pull it, replace it with a bigger drive, hot-enable it and have it sync. *Then* I was able to do the same with the original drive, giving me two bigger drives in the machine.
For the final trick, I was able to online resize the software raid partition and ext3 filesystems on the device, the whole time leaving the machine up and running.
The lesson here is: never trust anecdotal advice, especially from goofballs on slashdot that think they know everything.
If you have a requirement ("we need hotswap"), test the requirement and set your operational plans based on the test results. Not that hard...
Also, and back on topic, the software raid driver has a couple new modes, raid10 (similar to raid1+0, but with some extra features) and raid6 (essentially raid5 with an extra synced up spare). Those bear looking into.
Additionally, in the raid1 and raid5 modes, the newest versions of the software driver have the ability to auto-fix the bad sectors that are so common on the newest consumer drives.
The last advice I can give is to never, ever, ever, ever forget that RAID keeps you running when things go bad, but it IS NOT a substitute for backups.
Back your data up. Test restoring it semi-frequently. If you already know that from experience, you'll at least appreciate why I'm yelling about it
It has little to do with the actual interconnect at this point, but...
The best drives are SCSI. Actually these days the best drives are SAS. You won't find higher performance/more reliable 2.5" drives with any other interconnect. 2.5" is the future it seems. It's not even that expensive anymore. 2.5" 15kRPM 36GB drives for around $200. 100+GB for $200 if you only need 10k RPM. You can't even get 15k SATA drives, and you're going to get the same capacity for the same price with SATA at 10kRPM, but you're going to get a 3.5" drive instead of a 2.5" drive.
At 10k RPM, the prices are equal per MB, but the SAS (serially attached SCSI) drives will be 2.5" and use less power. Plus, the SAS RAID array would be faster than the SATA RAID array.
Latency also benefits from many disks for the same reason.
Not for writes. All RAID increases write latencies unless you have a battery backed cache.
we went from a LSI MegaRAID 320-1 + 4-drive SCSI RAID config to an Areca 1170 + 1GB RAM + 24-drive SATA RAID. Every aspect of performance is up by big amounts -- throughput, latency, multi-user access.
Gee, you installed modern technology to replace something 4 years old and you got a huge performance boost? What a shocker.
We ended breaking the array into 3 8-drive volumes and mirroring 2 volumes against each other for more redundancy. One of these days, we'll upgrade to faster CPUs and retest a 16-drive volume.
You likely won't see a difference with faster CPUs. You're bottleneck is either your memory bus, your PCI bus, or your controller. Your controller is the likely suspect, and it has nothing to do with throughput. It can probably only process a certain number of IO operations per second, no matter how much data is in each one, and you're probably hitting that limit. With the right benchmark, there is no reason to believe your server couldn't push several hundred megabytes per second, (450MB/second if you've got a dedicated 133Mhz PCI-X bus for your controller), but your cheap SATA controller probably can't handle the IOPS. I'd be surprised if it could push 15,000 IOPS, and well, unless all your IOs are 32K, you're PC isn't the problem... Even if you are, your PCI-X bus is the bottleneck, not your CPU.
SATA is so much cheaper that it's easy to afford completely 2 or 3 redundant storage servers for the same price as a single SCSI array. We do this at work in fact.
That depends on how much performance you need.
With SCSI around $3-5 per gig, and SATA around 35-70 cents per gig
At 10k RPM, SATA and SAS prices are almost the same. At 15k RPM you can't even get SATA drives. If you're going to pay the higher price anyway, you should pick the technology with the superior controllers.
If you only need 7200RPM, by all means, use SATA.
I have a number of 1+ terabyte arrays made from IDE and SATA drives, and my experience has been that failures come quick and often. Each of the machines has fans blowing large amounts of air over every drive, and the rest of the machines are top-notch quality. Whether the arrays were made from Maxtor, Seagate, or Western Digital, most have at least one drive die within the first six months.
In fact, about a week ago, a Western Digital RAID-edition drive died in one of my arrays - and that particular set of disks had only been in use for a week or two.
Of the SCSI arrays I've built and maintained, I've had three or maybe four drives fail in 5 or 6 years. It's a tough call, because it would be nice to build these arrays from SCSI drives - but when you're talking about 3 or 4 terabytes in an array, the number of zeroes on the cost of a SCSI solution makes you a bit dizzy. Because of that, I've continued to use SATA drives - but in RAID 6, with one hot spare, and two extra drives sitting on a shelf, waiting for another to fail. That might be overkill, but in two years if they don't make the same model of drive, you don't want to replace an entire array because one drive went out on you.
steve
Oh, you're not stuck, you're just unable to let go of the onion rings.
Question about raid6 vs raid10. Given that a two drive failure can kill a raid 10 setup, but not a raid 6 setup, when and why would you recommend a raid 10 setup over a raid 6 setup?
Awesome furniture, accessories and cabinetry in Santa Rosa, CA: http://humanity-home.com/
You can get 10K RPM SATA drives from Seagate if you need SATA and want low-latency access. The Seagate SATA Raptor is just a re-packaged SCSI Raptor with a SATA interface.
"we went from a LSI MegaRAID 320-1 + 4-drive SCSI RAID config to an Areca 1170 + 1GB RAM + 24-drive SATA RAID. Every aspect of performance is up by big amounts -- throughput, latency, multi-user access."
;)
What's that, a cheap SCSI card and 4 platters is slower than an expensive SATA card with gobs of cache and 6x as many disks on $unspecified_workload? Truely shocking
I have yet to see a SATA RAID controller that is as stable as the SCSI RAID controllers. Even the latest generations have hangs, crashes and corruption that disappeared from the major SCSI RAID controller venders many years ago. There is a significant learning curve for writing good drivers and firmware, and they just are not there yet.
...
The name brand RAID subsystems have a level of stability and flexability above that of the SCSI RAID controllers.
You get what you pay for
So, you need to decide how important is your data? I wouldn't trust much to a SATA RAID controller yet.
Isn't it always a smart idea to turn your PC off before you reach your hands inside the case?
Yes, turn off your PC before opening the case.
A server, though, is not a PC. It's a set of rack (or cabinet) -mounted enclosures that are designed to be accessed while the power is on.
"I don't know, therefore Aliens" Wafflebox1
"its a quick trip to the store for another 160 GB seagate."
And you'll be makin' that trip too.
"Take whatever's cheapest. Buy two."
You work for the government, don't you?
No, Government would be:
"Take whatever's most expensive, even if it's not suited for the job. Buy four, use one."
There are some people that if they don't know, you can't tell 'em.
I'm coming to this conversation pretty late, so I hope somebody who cares gets a chance to read this.
We're 3 to 4x the size of turboflux's org and both our 2TB 320 LVD scsi DAS (direct attached storage) units started getting wiggy early last year. After looking at our options we ended up with a couple of ExcelMeridian's (SecurStor XRS, if you care) DAS units populated with Western Digital SATAII drives in a Raid 6 config and connected back to our servers via 320 scsi controllers.
Western Digital has had a line of enterprise SATAII drives with extra layers of testing to certify that they're good enough for RAID duty for awhile; I think Hitachi has a line of these as well now.
The new units have been in service for about 10 months now without even a hiccup and they came with almost 3x the capacity at about 25% of the price that we paid for their scsi predecessors. One of our techs pulled a drive out of the config as a test...it kept humming and immediately started blasting us with email alerts, just as we had it set up to do. With 2 drives missing from the 16 drive config, performance degradation was noticeable but it still recovered okay.
FWIW.
It may be slim, but I've had it happen. The problem turned out to be a bad batch of fiber channel tranceivers, but it made my life hell for a month and finally cost me the entire array before we figured it out.
Raid 0+1 has a number of problems relative to true raid-10 in terms of reduncancy:
First of all, there's the problem that if you do lose 2 disks at close enough to the same time (e.g. a bad batch), that you don't have enough time to replace the first before the second goes, then you've got almost a 50% chance that the second bad drive is going to break your data.
The second is that when one drive goes bad, it breaks the entire plex. This means that, if you've got a 1TB of data, you end up copying the entire 1TB when the replacement drive kicks in. This hurts both your redundancy and your performance for a lot longer than just replicating the data that was on the bad drive in true raid-10.
The one problem with doing raid-10 is that I don't know of any drivers that let you do that in hardware and across controllers... (i.e. the solution of putting mirror pairs on the same controller means that, if the controller dies, you've got a broken array).
So for best redundancy, you want:
.... . ....
ctrl1 . ctrl2
dska1 . dskb1
dska2 . dskb2
dska3 . dskb3
but you're almost guaranteed to do it in software.
For best perfomance, you can do some of the raiding in hardware (how, exactly, may depend on your application), but at the cost of either performance or redundancy.
One good tradeoff is RAID 51 -- raid-5 on the controllers and mirror in software. This means that you can, guraranteed, suffer 3 dead drives without losing data, and one dead drive means simply regenerating the replacemant drive on rebuild -- not the entire plex. At the same time, you have the advantage of offloading some of the work to the controller but you can still lose an entire controller (and a disk) and keep running.
Free Software: Like love, it grows best when given away.
Oh. I was under the impression that the question was about a home file server (which might be in a normal PC case, depending on how many hard drives he puts in there - one or two 250GB HDs would be just fine for home use, unless you're doing some real serious stuff).
www.linuxpenguin.net
This is so fake. You've been had by a slick Mac ad. You geeks can't see it? I thought there was brains before beauty in this club.
LrM
Other than that, I agree with a lot of what you say.
sigs are a waste of space
> SAS (serially attached SCSI) drives will be 2.5" and use less power.
I was under the impression that current 10k rpm 3.5" form factor drives actually have 2.5" platters inside. If this is the case, the SAS disks are unlikely to use less power.
> All RAID increases write latencies unless you have a battery backed cache.
I suppose that with a fast RAID1 controller there would be some increase in latency but it is likely to be negligible (he says, pulling it out of his butt since he hasn't actually tested it). Certainly RAID5 will be slow on writes, which is what you were mostly talking about.
I was under the impression that current 10k rpm 3.5" form factor drives actually have 2.5" platters inside.
I wasn't aware of that, but it makes sense.
I suppose that with a fast RAID1 controller there would be some increase in latency but it is likely to be negligible
It isn't negligible. Your write can't be considered complete until your data is on both disks. Unlike old slow SCSI drives, disks on the same bus don't syncronize their rotations anymore, so you have to wait for the worst case rotational latency of both disks. It's about a 25% latency hit on average for a single IO. For linear writes the effect is less, but if you're running a database with lots of random writes, it's a big deal.
High end arrays and controllers get around this with a battery backed up cache, which allows them to call the IO complete once it is in cache. Low-end controllers, almost all SATA controllers, and many on-board controllers (even on server boards) don't come with a battery backed up cache though.