Experiences w/ Software RAID 5 Under Linux?

Advice: Get lots of RAM by Anonymous Coward · 2004-10-30 10:44 · Score: 0, Informative

Writing large files to an eight-drive RAID-5 arary will be butt slow unless you have a LOT of RAM.

The idea is that in order to write data to any sector on one of the drives, the sectors from six of the other drives need to be read, all XOR'd together, and then the result written to the remaining drive.

In theory, this could be done simultaneously--read from all drives at once. In practice, Software RAID and ATA isn't so good at that kind of thing. (Good hardware RAID is a different story.)

So the idea is that those six reads will take a reasonable amount of time, every time there is a write. If you have a lot of RAM, and/or don't write really large files, it won't be a problem because all the data can be cached in RAM and the reading/writing involving the disks can be done later, at the OS's leisure. However, if you don't have a lot of RAM, or copy really big files, you'll have performance issues.

You may not notice this for a little while, until your array starts filling up, because some implementations (not sure about the Linux software one) optimize it so that they assume unused sectors are filled with a known value, so they don't actually read from drives where the sectors haven't been written to yet (they keep a big table in memory). This is a GREAT optimization. But over time, it will get slower and slower.

So my advice to you is to install a lot of RAM in this system, whatever the motherboard allows. At least one gigabyte, but preferably two or more.

Re:Advice: Get lots of RAM by mortonda · 2004-10-30 11:00 · Score: 5, Interesting

The idea is that in order to write data to any sector on one of the drives, the sectors from six of the other drives need to be read, all XOR'd together, and then the result written to the remaining drive.

Your logic eludes me. The blocks do not need to be read, as we are in the process of writing. We already have the data, because we are writing, so why would we re-read the data?

Furthermore, block sizes default to 4k, though you could go to 8k or 32k block size. At any rate, you don't need a gig of RAM to handle this.

Finally, XOR is not that expensive of an operation, and a 500Mhz CPU is going to be able to handle that faster that any but the most expensive controller cards.

So unless you are actually a RAID kernel developer, I don't buy your story.
Re:Advice: Get lots of RAM by Anonymous Coward · 2004-10-30 11:05 · Score: 5, Informative

Another piece of advice would be, since you have eight identical drives, to use only seven drives in the RAID array, and keep the eighth one out of the array entirely, either outside the computer in an antistatic bag or as a "hot" spare--installed but idle.

When one of the drives fails--and one of the drives will fail--this will allow you to swap in the replacement drive immediately, before another drive fails. (Remember, if two drives fail in a RAID-5 array, you lose data.) You can then return the defective drive, get a replacement from Maxtor, and when that one arrives FedEx in a few days, that one will be your new "spare."

You can either keep your spare drive unused, outside the computer, or keep this spare "hot"--in the computer, connected and ready to go, but unused by the array or anything else, and have the array fall over to it automatically when a drive fails.

Both ways offer advantages. If you keep the drive out of the computer, since you need to shut down to remove the bad drive, you can install the spare drive at that time. If you were to keep the drive "hot" in the meantime, your extra "new" drive has been spinning for months or years, and exposed needlessly to heat. Which increase its probability of failure, making it essentially as likely to fail as all your other drives that have been running the whole time.

However, keeping the spare "hot" means that the array can be rebuilt sooner, in some cases automatically before you know there is a problem. This can reduce the possibility of data loss. You will have to reboot twice--once to remove the defectie drive to return to Maxtor, and once when the replacement arrives to install it as the new hot spare.

Which of those two choices is a judgement call, but it's absolutely critical to have a spare drive on hand.
Re:Advice: Get lots of RAM by Anonymous Coward · 2004-10-30 11:05 · Score: 0

Hmm, nope, I think that a 3ware controller with its risc processor can push more XOR operations but I agree with you on the amount of ram situation. Why??
Re:Advice: Get lots of RAM by Jah-Wren+Ryel · 2004-10-30 11:09 · Score: 1

If you copy (or just write) really big files, then there should be no reason for the raid-5 driver to read all the other sectors in the parity chunk (don't know official terminology for it offhand) since, if done right, you will be writing all of the sectors with brand new information anyway.

--
When information is power, privacy is freedom.
Re:Advice: Get lots of RAM by Anonymous Coward · 2004-10-30 11:09 · Score: 4, Informative

The idea is that in order to write data to any sector on one of the drives, the sectors from six of the other drives need to be read, all XOR'd together, and then the result written to the remaining drive.
Um. No. Not if the RAID5 implementation is reasonably sane. Assuming all the drives are in good working order, all the software has to do is read the original block off the drive; the parity block off the appropriate drive; XOR the two values together, and XOR the new data; and you have the new parity block.
IOW: Two reads, and two writes. Not six reads and two writes. But yes, large amounts of RAM is a good idea. Of course, if a drive goes south, everything goes out the window and your performance will be shot until you replace the dud drive and everything resyncs.
Re:Advice: Get lots of RAM by GigsVT · 2004-10-30 11:11 · Score: 2, Interesting

I agree, the parent post is just a troll to see how gullible the moderators are. Apparently he proved his point. :)

--
I've had enough abrasive sigs. Kittens are cute and fuzzy.
Re:Advice: Get lots of RAM by Anonymous Coward · 2004-10-30 11:17 · Score: 3, Interesting

Your logic eludes me. The blocks do not need to be read, as we are in the process of writing. We already have the data, because we are writing, so why would we re-read the data?

That would depend on the nature of the write. If you're writing the initial data it's unlikely that you'll require reading. However when you go to update the date you may have to perform reads in order to calculate the parity required for the update.

Software RAID 5 is very reliable but does suffer a performance hit. Not because of the XOR computations like many here are suggesting. It's because each logical write needs to be translated into physical reads/writes...which consume time.

The beauty of software RAID, at least software RAID implementations such as Veritas, is that it allows you to spread the RAID across a number of controllers.

Listen to this guy...he knows more than the others who consider the XOR computation the slow link in software RAID 5. It's not.
Re:Advice: Get lots of RAM by Anonymous Coward · 2004-10-30 11:19 · Score: 0

One other thing...you don't need a whole lot of RAM. That's one suggestion I wouldn't follow.
Re:Advice: Get lots of RAM by Anonymous Coward · 2004-10-30 11:22 · Score: 5, Informative

Your logic eludes me. The blocks do not need to be read, as we are in the process of writing. We already have the data, because we are writing, so why would we re-read the data?

Unless you write across a whole row in the array, how are you going to compute the new parity without reading in something? This is the "small write problem", and it is why expensive RAID controllers have a non-volite writeback cache.

The current kernel does read in the whole row to recompute the parity for simplicity. Technically, though, you just need to read in the block you are modifying and the parity block, making writes take 4 operations under RAID 5, but unless something has recently changed, Linux doesn't do that. A gig of RAM, however, will allow a degree of volitile write-back cache, to help offset what will otherwise be poor write performance.
Re:Advice: Get lots of RAM by Alien+Being · 2004-10-30 11:23 · Score: 1

"Your logic eludes me. The blocks do not need to be read, as we are in the process of writing. We already have the data, because we are writing, so why would we re-read the data?"

It depends on the nature of the updates. If you're writing long contiguous chunks (e.g. file copy), then you're right, the blocks are already resident. But if you just want to update a single block (database update), it might involve reading from the other drives.
Re:Advice: Get lots of RAM by mortonda · 2004-10-30 11:23 · Score: 1, Informative

Um, no. Since we are writing, we already know what the data is. Just write. No reads.

RAM is helpful on ly in the sense that it can cache data and make things appear faster if the data is already available in cache. It won't really help it read/write data any faster.

If a drive goes out, write performance is inchanged, as the XOR operation must be done no matter what.

Read performance depends - it's just an XOR operation, which is not very difficult. A 500Mhz CPU will still be mostly idle even in degraded mode. Some implementations *could* do the XOR in regular mode too, to check for data errors, in which case, no performance is lost in degraded mode anyway.
Re:Advice: Get lots of RAM by aaronl · 2004-10-30 11:26 · Score: 4, Informative

There's already some issues here because these are eight identical drives bought at the same time. They are very likely manufactured by the same machines in the same factory at the same time. This increases the liklihood of multiple disc failures for a variety of reasons. This is why many admins will replace discs on a regular schedule. For example, you buy your eight shiny new discs. Then you run them for a year, and replace one, a few months later you replace another, and so on. Then every two years from then you replace that disc again.

Also, any half-decent RAID implementation will have that hotspare in the machine with its spindle off until it is needed. So it won't have been spinning for months/years at all. Not quite as good as having it in a box as far as wear and tear, but very close.
Re:Advice: Get lots of RAM by Avakado · 2004-10-30 11:30 · Score: 1

If the smallest block you can write is 512 B (a not very unlikely restriction), an attempt to write 511 B would mean you have to read what's already there if you don't want to garble a byte.

--
The world will end in 5 minutes. Please log out.
Re:Advice: Get lots of RAM by dbullock · 2004-10-30 11:41 · Score: 2, Informative

So my advice to you is to install a lot of RAM in this system, whatever the motherboard allows. At least one gigabyte, but preferably two or more.

There's a reason this is posted anonymously. It's absurdly incorrect. Disregard and move on.

--
http://www.bullnet.com
Re:Advice: Get lots of RAM by mortonda · 2004-10-30 11:43 · Score: 1

Ah ok, I see what you are saying... although, chances are that the whole sector has already been read previously and is already in cache anyway.

For example, when the data was read prior to being modified.

The only time this would be a problem is when a whole new file is being written. Even then, the sector may be on the end of a free space inode chain, and so the kernel know no other data is on that sector, and it can do what it wants. Depends on the filesystem, I guess.
Re:Advice: Get lots of RAM by Kent+Recal · 2004-10-30 12:00 · Score: 1

Bullshit.

RAM is not an issue at all, you can go with 256M or less and be fine.
You'll likely take a performance penalty from the IDE bus though because when writing to the array your data is sent to n drives instead of one. Caching helps here but if you're paranoid about power outages you'd better turn that off. Other than that linux softraid is perfectly fine and quite a lot people use it in production systems. If you need better performance you'll have to spend on SCSI or one of these hardware-raid IDE cards or a bit of RAM for caching but don't expect wonders from the latter...
Re:Advice: Get lots of RAM by wmeyer · 2004-10-30 12:02 · Score: 1

Writing large files to an eight-drive RAID-5 arary will be butt slow unless you have a LOT of RAM.

Not unless Linux has a much worse implementation than MS -- I've had extensive experience with it there, and the performance is quite good.
The idea is that in order to write data to any sector on one of the drives, the sectors from six of the other drives need to be read, all XOR'd together, and then the result written to the remaining drive.

Um, not quite. Some review of RAID levels would be in order. RAID 5 does impose read/modify/write overhead, but there is not a dedicated parity drive -- parity is striped.

My first question to the original poster, however, would be what controllers he plans on using. By the time he connects 8 drives, using N controllers, he's a short distance from what a 9508 from 3ware would have cost.

--
--- Bill
Re:Advice: Get lots of RAM by Anonymous Coward · 2004-10-30 12:03 · Score: 1, Informative

Consider one row in a 7 drive array. It has 6 data blocks and one parity block:

D1 D2 D3 D4 D5 D6 P1

where P1 is the XOR of D1 through D6.

If I write to D1, but leave any of D2-D3 alone, then it is necessary to read SOMETHING in order to calculate the new parity. Yes, I know what I'm writing, but unless I overwrite the whole thing, I must perform extra operations in order to update the parity block correctly. These extra operations degrade performance, and are known as the small write problem. As another AC above said, the update can be done with two reads and two writes; read the old D1 and the old P1, then write the new D1, and write P1 to be (old D1 XOR new D1 XOR old P1). It's a bit of trickery, but it does give the correct parity block. It does, however, take two reads and two writes, to update the one block.

Linux (last I looked) doesn't do this. Instead it takes the simpler approach of reading the blocks in the row that it isn't updating (D2 through D6 in this case), and then computing P1 as the XOR of D1 through D6 again.

The small write problem is a big deal. Although the IOs can happen in parallel, the latency for the write becomes the maximum of the reads plus the latency of the parity write. The larger number of IOs also keep the array busy when it could be doing other things, which degrades the performance of those other operations. And it causes this performance degregation for small updates (those under the stripe size), amoung the most common operations. Even if all of your files are big, and written in a streaming manner, the metadata updates are generally in a different row in the array, and are small, isolated writes. A journaled file system, depending on how it is implemented, can be much worse for generating lots of scattered writes. Here is a paper from CMU that gives one possible solution (one that isn't implemented by Linux). The traditional solution is write caching--you delay the write until either you've updated the other entries in the row, you've read the other entires in the row, or it is otherwise convienient to do the update (i.e. array isn't busy). This is of course dangerous because your data isn't on disk but in RAM. OTOH, witha good UPS, loosing the contents of RAM is a relatively rare event. To sidestep the volitility of RAM entirely, nice HW raid controllers have some amount of non-volitile memory (either NVRAM or battery backed DRAM) for this purpose. Writeback caching can also help perfomance on non-RAID devices, since it allows you to reorder the writes to minimize head seeks and rotational latency. These two, especially the head seeks, are what make disk IO slow.

You obviously don't have much background in storage. Try reading Chen's classic paper on RAID, go and search for a few papers the reference it, and then come back and spout off. Until then, quit giving people bad advice.
Re:Advice: Get lots of RAM by ezzzD55J · 2004-10-30 12:03 · Score: 1

" Um, no. Since we are writing, we already know what the data is. Just write. No reads."
Presumably, we are talking about the parity disk here, otherwise there aren't any reads involved at all. In that case, you need to read the old parity in order to compute the new one.
Re:Advice: Get lots of RAM by ezzzD55J · 2004-10-30 12:06 · Score: 1

"If you copy (or just write) really big files, then there should be no reason for the raid-5 driver to read all the other sectors in the parity chunk (don't know official terminology for it offhand) since, if done right, you will be writing all of the sectors with brand new information anyway."
Usually true, but I think the GP is talking about updating the parity block when just one block is updated. Still, all other reads aren't necessary as the parity block is updated (XORred) with the XOR of the old and new data block.
Your case is even better of course, if the block device logic is smart enough to group transactions like these together (safely), which involves (safely) waiting to see if any 'close' transactions will be happening..
Re:Advice: Get lots of RAM by ezzzD55J · 2004-10-30 12:13 · Score: 1

This post is full of nonsense (see replies below, some of mine)
Re:Advice: Get lots of RAM by barc0001 · 2004-10-30 12:48 · Score: 2, Informative

For a little more (well, maybe more than a little) than the amount of coin a lot of RAM will cost you, go get a 3Ware 8 port RAID card instead. I run one, and it kicks ass. I see the one we got (Escalade 7506-8) on Pricewatch for $366. The RAID is fast, fault tolerant, and has a little web interface to let me know its status. I've currently got the drives configured as 7 in a RAID 5 with the 8th as a hot standby. I am very pleased with it.
Re:Advice: Get lots of RAM by SealBeater · 2004-10-30 12:49 · Score: 2, Informative

Just to insert my 2 cents into this, I have a 4 disc 750 raid 5 SATA array, on
a PIII with 128 megs of ram. LVM on top of the array, and I have never run
into a problem with serving files via NFS or SMB. More ram is always nice, of
course, but again, I have not ran into any problems.

SealBeater

--
-- Its survival of the fittest...and we got the fucking guns!!!
Re:Advice: Get lots of RAM by noda132 · 2004-10-30 12:54 · Score: 1

The only time this would be a problem is when a whole new file is being written.

That's the second-most-common operation on file servers (the actual reading of files being the most common).
Re:Advice: Get lots of RAM by dspeyer · 2004-10-30 14:38 · Score: 3, Informative

One other thing that's critical is to monitor status carefully. One of the points of RAID is transparent failure recovery, but the Linux version is too transparent: you can lose a drive and not notice it at all, until a second drive goes (third if you've set up a hot-spare as described) and then you're in trouble.
Probably the best move is to have a cron job examine /proc/mdstat and e-mail you if it's troubled.

--
Sig:Why copyright isn't a fundamental human right
Re:Advice: Get lots of RAM by jdibb · 2004-10-30 15:31 · Score: 3, Informative

You don't have to read the data from ALL the other drives to do a Raid 5 write, but you do have to read the data that's being overwritten, and you do have to read the parity that corresponds to the data that's being overwritten. Unless, the data you are writing on a specific IO spans most or all of the data drives, then it might be cheaper to read the data that you are not overwriting and create a whole new stripe of data. I AM (and have been for 13 years a RAID developer), but not for LINUX.
Re:Advice: Get lots of RAM by Cramer · 2004-10-30 15:54 · Score: 1
- The idea is that in order to write data to any sector on one of the drives, the sectors from six of the other drives need to be read, all XOR'd together, and then the result written to the remaining drive
That's not RAID 5. That's RAID 3 or 4. 5 is striped data AND parity. What you are describing has a single parity drive. (And yes, it's pretty damned slow.) With the additional caching effects, if you breifly delay writing, both blocks of a parity set may get written at the same time, so zero block reads, 3 block writes. Normally, there will be 1 block read and 2 blocks written.
Re:Advice: Get lots of RAM by apdt · 2004-10-30 15:57 · Score: 3, Informative

Probably the best move is to have a cron job examine /proc/mdstat and e-mail you if it's troubled.

Or you can just have mdadmd (pard of the mdadm suite.(comes with my distro (SuSE 9.1))) running, and it'll monitor your raid arrays, and email you when there's a problem.

--
I lay awake last night wondering where the sun had gone, then it dawned on me.
Re:Advice: Get lots of RAM by mortonda · 2004-10-30 16:18 · Score: 1

The advice I was trying to give was to not discount Software RAID on account of RAM or CPU - it sounds like plenty for that arrangement. It also sounds like they are not building the very best system, so most if this wont really have any relevance.

I hadn't considered the fact data smaller than the stripe size, but before you pull out your flaming gins look at the parents of my posts to see how far off they were... Software RAID is perfect for the OP's problem, no need to use hardware RAID and Linux is most certainly up to the task.

Since you seem to know so much... isn't it likely, though, for many operations, the data is already in cache? For update operations anyway, the data was probably read recently, which would have had the entire block. In this case, extra reads may not be necessary.
Re:Advice: Get lots of RAM by k12linux · 2004-10-30 16:23 · Score: 1, Informative

Your logic eludes me. The blocks do not need to be read, as we are in the process of writing.
Unless you write across a whole row in the array, how are you going to compute the new parity without reading in something?

Everything you need to write is already in RAM except the checksum black. So if you have a 7-drive RAID5 array, the RAID subsystem can take 6 blocks of data, compute a parity block from them then write one block to each drive. It's not like it is going to write random sized chunks of data and can not tell what is going to be written without actually writing it to disk.
Even hardware RAID cards typically don't have a lot of RAM. They also don't write to the drives, re-read what they have just written and create a parity bit from it. Neither does software RAID in Linux.
The only time the system should need to read in order to generate a block is when it is rebuilding after a drive has been replaced.
Re:Advice: Get lots of RAM by darqchild · 2004-10-30 16:35 · Score: 1

Ram isn't the issue.
The bottle-neck in Software RAID is not the RAM, the CPU, the extra math that needs to be done, it's the the PCI bus. In hardware RAID, you send the data to the card, and it distributes it among the attached disks. In a software RAID 1, you need to send the data across the bus once for each disk. In a hardware RAID you need to send the data only once. So far, in a soft RAID 1 with 2 SATA disks, i've not noticed any significant bottleneck.

Can you sacrafice a little bit of speed for some cheap reliablity?

--
What? Me? Worry?
Re:Advice: Get lots of RAM by captaineo · 2004-10-30 16:56 · Score: 1

RAID stripes are typically on the order of 64KB-128KB. If you just write one filesystem block (4KB) you have to read the entire rest of the stripe to figure out the parity. But, modern drives are pretty good at large contiguous reads, so it's not that much of a big deal.

You are correct that modern CPUs kick the ass of most (probably all) RAID controllers at performing the XORs. The reason you'd want a hardware RAID system is for saving CPU cycles, cache pollution and memory/bus traffic, not for raw XOR speed. (in software RAID-5 you will see quite a bit of CPU time taken by the RAID computations, vs. virtually nothing with hardware RAID)

And yeah I'd go for as much RAM as possible in a file server.
Re:Advice: Get lots of RAM by Anonymous Coward · 2004-10-30 17:06 · Score: 0

Give this guy a break. All of you jumped on him like a bunch of hawks ready for the kill. So what he made a few errors. If you can't answer his question then shut the hell up. If you would like to correct him and answer his question, great. Get a fucking life.
Re:Advice: Get lots of RAM by drwho · 2004-10-30 17:08 · Score: 1

I dunno what he's talking about. I built a really nice system with 4 SATA-150 drives, 384MB of ram. Is that a lot? And it worked fine. But make sure you pay attention to cooling. I used a generic case and when the AC went out in the server room, it only took half an hour for a drive to fail! After replacing it, I made sure there was enough cooling and everything worked fine, even when the AC failed again, and was off overnight.
Re:Advice: Get lots of RAM by ThePiMan2003 · 2004-10-30 17:52 · Score: 1

Assume 4k blocks. I write 4k all in one block, now I need to read the other disks so I can computer the checksum to write to the lsast one. I only wrote 4k so I only have 4k in ram.
Re:Advice: Get lots of RAM by Quixote · 2004-10-30 18:04 · Score: 1

You don't have to read in all the six sectors[sic].
Firstly, you aren't dealing with sectors; you are dealing with blocks. But we'll use your terminology and stick to "sectors" for this discussion.
Assuming you start with blank sectors, you don't have to worry about the sectors corresponding to the stripe which are empty. Why? Because 0 ^ 0 is still 0. So, no dreaded reading of the 6 sectors in this case.
Now, if you are updating a sector that has already been written, then you need to just read the parity sector(P) as well as the contents of the sector being overwritten (O). Assume the new data is (N). Then, your new parity sector becomes P' = (P ^ O ^ N). So, what would have been just a write gets translated into 2 reads (of P and O), an XOR (trivial) and 2 writes (of N and P').
Re:Advice: Get lots of RAM by k12linux · 2004-10-30 18:13 · Score: 1

Assume 4k blocks. I buffer up 24k (6 blocks) calculate a 4k parity block and write 7 blocks to the 7 different disks.
Lets say that so little disk activity is going on that it needs to flush buffers before it has buffered 24k of data. (I'm making an assumption how it really behaves now) I would expect it would calculate the XOR across as many 4k blocks as it had and write some blocks as blank. When it went to write more data it almost certainly has those last few 4k blocks still cached in RAM.
It should be extremely rare that it actually has to go back to disk to read data for creating the parity blocks.
Re:Advice: Get lots of RAM by arivanov · 2004-10-30 19:04 · Score: 2, Interesting
I have about 3 TB on 3ware and 2TB on linux software RAID. Most of it RAID5. My recommendation is - stay away from 3ware unless you know what you are doing. It is a very nice controller, but it extremely fussy as far as bus noise levels are concerned. Also, it does not support PCI parity so you have no real indication of what, how and where fails. When used with riser cards we had to fit additional 33Mhz cards to drop the PCI down to 33 and retrofit a serious amount of extra grounding to get machines stable. Overall, a chassis with risers for 3ware is a no-no. You are better off with a 3U+ chassis where cards are straight on the bus. This is under any OS - just read the threads on BSD-stable.
Linux software RAID5 has a considerably better chance to work nowdays. There are very few controllers out there that have unresolved bugs. Off the top of my head here are a few:
- Serverworks - any kernel version. Has seriousperformance problems with slave drives. Kernels before 2.4.23 die at random.
- CMD646/9. Used to be the best controller out there, unfortunately no more. It supplies bogus information for the ACPI tables so it is no longer useable on SMP as of Linux 2.4.23 and later. IRQs do not get initialized.
- Promise - I personally stay away from them as many are not supported properly.
As far as controller duty roster is concerned we should also mention Via. From being the worst controller for Linux once upon a time in 1997-9 it has become the best. I have been getting better IO performance on ITX with C3 then on Xeons with server controllers for some time now (starting from around 2.4.23).
--
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
Re:Advice: Get lots of RAM by superpulpsicle · 2004-10-30 20:46 · Score: 1

Stay away from hardware RAID at home that's my advice. Software RAID has the luxury of always being able to see the filesystem while you move data.
Re:Advice: Get lots of RAM by Fallen_Knight · 2004-10-30 21:03 · Score: 1

More ram is always a good thing:)
Re:Advice: Get lots of RAM by Anonymous Coward · 2004-10-31 00:38 · Score: 0

If you just write one filesystem block (4KB) you have to read the entire rest of the stripe to figure out the parity.
No, you don't.
All you need to read is the parity block and the block being written.
The new parity can be figured as follows:
new-parity-block = old-parity-block XOR old-data-block XOR new-data-block
Re:Advice: Get lots of RAM by Anonymous Coward · 2004-10-31 01:03 · Score: 0

I don't know how linux does it, but you don't have to read the whole row, only the parity block. If you XOR the block you want to write with the parity block, you get a new parity block for the whole row. That makes 1 read and 2 writes.
Re:Advice: Get lots of RAM by Anonymous Coward · 2004-10-31 01:06 · Score: 0

Additional note: This is also why raid 5 (distributed parity) makes sense, when you are writing multiple blocks, reading/writing the parity will be distributed across disks.

Think about it, what's the point in distributing the parity if you read the whole row every time. Might as well use raid 4.
Re:Advice: Get lots of RAM by Jennifer+E.+Elaan · 2004-10-31 02:38 · Score: 1

We just use Nagios. We had a drive failure recently, and we got paged immediately letting us know the drive had failed. It worked exactly as we expected.

--
Hardware, software, and blinking lights!
Re:Advice: Get lots of RAM by k12linux · 2004-10-31 05:54 · Score: 1

You are absolutely right. I hadn't considered that. An XOR of the parity block would provide the same results as XORing all current data blocks against the new one.
Re:Advice: Get lots of RAM by LWATCDR · 2004-10-31 11:11 · Score: 1

Or do both.
only use six drives live in the raid and keep on as a hot swap and one as a spare in a box. I would suggest that if you want maximum reliability I would go for a hardware raid card.

--
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
Re:Advice: Get lots of RAM by dbIII · 2004-10-31 13:14 · Score: 1

or keep this spare "hot"--in the computer, connected and ready to go, but unused by the array or anything else
Not always a good idea - if your RAID disks die due to heat (seen that several times) it's best if the spare has not been exposed to the same heat. If you have a lot of disks in the same confined space (ie. badly designed PC case with too many disks) and a string of air conditioning failures you can find that you will be replacing drives one after another.
Re:Advice: Get lots of RAM by Anonymous Coward · 2004-11-01 02:04 · Score: 0

Wrong.

Parity = sum(data blocks)

New parity = old parity - old data + new data

Where + and - are the same XOR operation

So new parity takes 2 reads and 2 writes.
Re:Advice: Get lots of RAM by vlm · 2004-11-01 02:23 · Score: 1

Whats your backup plan for the raid card?
Five years after the hardware card is discontinued I'll still be able to run my software raid...

--
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
Re:Advice: Get lots of RAM by Anonymous Coward · 2004-11-01 04:45 · Score: 0

> Also, any half-decent RAID implementation will have that hotspare in the machine with its spindle off until it is needed. So it won't have been spinning for months/years at all. Not quite as good as having it in a box as far as wear and tear, but very close.

OK, so is the Linux software RAID half-decent or not? Does it support having a spare drive on an idle spindle?..
Re:Advice: Get lots of RAM by k12linux · 2004-11-01 05:08 · Score: 1

No. That is incorrect. First of all XOR is NOT the same as + and -. See this explanation to brush up on XOR and RAID parity.
Here is an example:
First if three data blocks and one parity are written at once: 1010110010011 - data 1100101101100 - data 0111011010110 - data ------------- 0001000101001 - parity (XOR of all data) (Four block writes.) Next, if two data blocks and parity are written then later a third data block and new parity is written. 1010110010011 - data 1100101101100 - data ------------- 0110011111111 - parity (XOR of original data) (3 block writes) 0110011111111 - parity (possibly read from disk) 0111011010110 - data ------------- 0001000101001 - parity (Note same result.)

One disk read and two writes... exactly as grandparent poster said. No need to read any of the original data blocks. Only the original parity block. Even then only if it isn't still cached in RAM which is quite possible.

Please! by Anonymous Coward · 2004-10-30 10:44 · Score: 0, Insightful

Do yourself a favour and buy some more or less cheap hardware RAID controllers. You won't regret it. Software RAID is nothing more than "showing it's possible".

Re:Please! by Anonymous Coward · 2004-10-30 11:01 · Score: 0

Yup, I agree. I have the same system but I use a hardware controller (3ware 7506) and I would never go back to software raid.
Re:Please! by Anonymous Coward · 2004-10-30 11:05 · Score: 0

I'll agree. About a month ago, the local CompUSA had 240 gig drives on sale for $120. I bought 3 expecting a long-term software raid5 solution for all my storage needs. In the process of copying files over to the raid after setup, it started butchering inodes.
There were certain directories on the raid that couldn't be read (or removed). This was quite recently on a p4 machine (2.6.8 kernel, i think).
Can't say this is the norm, but after loosing a few fine albums to The Great Raid Migration Of '04, I haven't been able to trust linux software raids.
Re:Please! by Gherald · 2004-10-30 11:06 · Score: 5, Informative

> Do yourself a favour and buy some more or less cheap hardware RAID controllers. You won't regret it. Software RAID is nothing more than "showing it's possible".

There is no such thing as a "cheap" hardware RAID 5 controller. Well there is, but they'll still set you back at least $120 and are crap.

There are RAID controllers from highpoint and promise, et al that are card-based, but they are still CPU bound (that is where the XOR really takes place). So they're really nothing more than a controller with a driver that does the calculations in the CPU. These cards are good for booting windows to a software RAID (since that is essentially what they are) but not good for anything else.

Most motherboards especially those with only 2 RAID ports (whether IDE or SATA) are software-based, as well. The nvidia nforce3 250 is one of the few notable exceptions.

But the bottom line here is: Linux Software RAID 5 is a logical approach if simple redundant mass storage is your main concern, and will save you at least $120. Also note that for RAID 0/1 it doesn't really matter if you go hardware or software since they aren't very processor intensive anyway. Pure software RAID 0/1 seems to be easier to set up in Linux (less mucking around with drivers) so it often makes sense to go with it for that reason alone.

--
The unofficial /. digest
Re:Please! by Anonymous Coward · 2004-10-30 11:17 · Score: 0

Absolutely incorrect. Promise SX4/SX4000 provides hardware XOR capability on the controller. XOR'ing never has to touch the host processor.
Re:Please! by Gherald · 2004-10-30 11:19 · Score: 1

> Absolutely incorrect. Promise SX4/SX4000 provides hardware XOR capability on the controller. XOR'ing never has to touch the host processor.

Dude, the SX4 will run you $160. I was talking about the cheap sub-$100 promise (and highpoint) cards.

--
The unofficial /. digest
Re:Please! by Anonymous Coward · 2004-10-30 11:40 · Score: 0

well i was talkign about more or less cheap controllers. in the arena of raid controllers 160USD is fucking cheap!
Re:Please! by phrasebook · 2004-10-30 11:58 · Score: 1

So you stuffed up (didn't create it properly). That says nothing about Linux software RAID. It's just as easy to break a 'hardware' RAID, and what happens when the RAID hardware itself breaks?
Re:Please! by ErikZ · 2004-10-30 12:01 · Score: 1

More than that. You'll need to buy cache memory for it.

--
Democrats or Republicans. They are both taking us to the same place and they are not afraid of us anymore.
Re:Please! by Gherald · 2004-10-30 12:02 · Score: 3, Interesting

> well i was talkign about more or less cheap controllers. in the arena of raid controllers 160USD is fucking cheap!

Not compared to $0.

You see, the typical budget RAID 5 builder just wants to store his collection of MPEG4s, MP3s, and other downloads or perhaps uncompressed hobbyist video. It's not a database, it's not a 150+ employee corporate file server, it's just personal. Performance is not a concern.

And if performance is a concern (say he wants / on these disks) then the cheap way to go is software RAID 0, 1 or 1+0 (aka 10) *COMBINED* with a RAID5.

For instance, I just built myself a new system with four 300gb drives and partitioned each one like so:

50mb - /boot
1gb - swap
20gb /
5gb - /tmp and /var /home

For the 50mb, I made a bootable RAID 1 of four drives (grub can boot this, dunno about lilo)

For the 1gb swap, I made a RAID 1 with two drives and a RAID 1 with the other 2. Thus I have a net of two 1gb swap partitions, with redundancy so my system will never crash due to drive-induced paging errors. This is essentially a RAID 0+1, though I let the kernel's swap system handle the RAID 0 aspect by giving them equal priorities.

For the 20gb /, I did the same thing (pair of RAID 1s) and put a RAID 0 on top of that, for a net of 40gb redundant and fairly speedy storage.

For the 5gb /tmp and /var I made a simple 10gb RAID 0 for each. Not a whole lot of need for redundancy here, I make a point of backing up the important /var stuff.

With the four equal-sized partitions that were left, I made the RAID 5 for /home

Don't you see what a great cost-effective approach this is?!?

Maybe you work for some company with plenty of money lying around for $160 RAID controllers. But I'm in business for myself, and I don't see the sense in spending money where it isn't needed. Besides, the flexibility of software RAIDs (per-partition, not per-drive) would be well worth it to me even if something like the SX4 were cheaper.

--
The unofficial /. digest
Re:Please! by k12linux · 2004-10-30 16:36 · Score: 1

It's not a database, it's not a 150+ employee corporate file server, it's just personal. Performance is not a concern.

I have heard from couple of Dell server techs and a Red Hat employee that in their testing performance of software RAID outperformed hardware. Their explanation was that in all but the most expensive controllers, the card's CPU speed was much slower than you might assume... especially compared to a fairly modern PC's CPU.
The Dell guys said that most hardware vendors don't like to mention that because it cuts into sale of the cards.
Re:Please! by flsquirrel · 2004-10-30 16:52 · Score: 1

If the machine is a dedicated file server this is conceivable. But in that case, you're wasting your time worrying about performance on the array anyway since your network will be a super bottleneck anyway. In this situation, the only purpose of raid 5 is for data integrity.

The real place where the performance side of raid 5 matters is for things like application (especially in the web sense) and database servers. That is, where the data consumed by the server is far more significant than the data dispatched to the clients. In cases like this, CPU cycles are generally of high value and any ability to offload work to other devoted hardware is a necessity.
Re:Please! by Fallen_Knight · 2004-10-30 21:37 · Score: 1

i'm curious, i've been looking for a hardware raid solution, what are your thaughts on these cards:

http://www.3ware.com/products/serial_ata9000.asp

~500 USD for the 8 port one though=/
Re:Please! by Gherald · 2004-10-31 00:11 · Score: 2, Informative

3ware raid controllers kick ass, they are the best on the market especially for Linux

frickin expensive, though... if you need that kind of performance it'd probably be speedier and more cost effective to do a software RAID 0+1

--
The unofficial /. digest
Re:Please! by swilver · 2004-10-31 00:28 · Score: 1

That's ridiculous. The only advantage a hardware controller offers is reduced PCI bus and CPU usage. If you however have a dedicated server, that doesn't have anything else to do, software raid is perfectly acceptable and can easily perform as well as hardware raid.
I run a 6 drive software raid, and put encryption on top of it, and I STILL get 10-15 MB/sec performance... plenty to feed a 100 mbit HOME network.
Re:Please! by swilver · 2004-10-31 00:31 · Score: 1

Looks like you didn't quite get it installed properly. I'm running software RAID5 with 6 drives on a Linux 2.6.7 kernel (have been for over 6 months, migrated kernels several times now). None of the data is damaged, not even during a rebuild of the array when I had to replace a drive.
Re:Please! by Anonymous Coward · 2004-10-31 03:03 · Score: 0

I've always found that the system CPU is less efficient at maintaining the RAID5 than a dedicated drive array processor. That's probably become less of an issue with the faster CPUs these days. It's been a while since I've bothered with RAID5 since drive space got so much cheaper that I can just mirror everything.
Re:Please! by Jennifer+E.+Elaan · 2004-10-31 03:12 · Score: 1

I'm not a server tech, but I am an EE, and I wouldn't find that even remotely surprising. The XOR engines on hardware RAID cards operate at core clocks no faster than a couple hundred MHz (simply by virtue of the fact that they are made with inexpensive CMOS processes). Now, they can benefit to a certain extent from parallelization, but modern CPUs with SIMD instructions can make the same statement.
About the only time now that a hardware RAID card makes sense is when the number of drives and total thoroughput would saturate the PCI bus. The hardware RAID card would be a lot less likely to encounter this situation, since it uses only internal busses for drive communication.
That said, hardware RAID cards generally have only a very limited cache RAM onboard, and a properly designed software RAID that integrates with the block cache could potentially make big wins in performance in systems with lots of RAM.

--
Hardware, software, and blinking lights!
Re:Please! by tigersha · 2004-10-31 09:33 · Score: 1

Been there with a Mylex DAC 960 and believe me, you don't wanna be there. In a server with redundant CPU's, redundant PSU and redundant disks trust the only non-redndant thing, the RAID card, to go pop.

The Mylex stores its RAID stuff on the card itself (not on the disks, as it should) so if the card is screwed, and your people never wrote down the friggin config, you are well, screwed. On a real RADI card the array config is stored in the first few sectors on the disk so when the card goes, you stick the disks on to the new card and you are just fine.

Fortunately we got another old Mylex on EBay and switched the NVRAM chips and it worked....one of the worst days of my life. And I was never happy to see NT 4.0 boot until that day.

Oh well, working for a non-profit tends to be an adventure. My servers stands on bricks to protect them from the occasional flooding.

--
The dangers of excessive individualism are nothing compared to the oppressiveness of excessive collectivism
Re:Please! by stanmann · 2004-11-01 08:48 · Score: 1

I got my sx4000 for under 100 and that was over 2 years ago.

--
Food not Bombs is a nice platitude but it breaks down when you notice that the Bombees are usually well fed

Works great by AIX-Hood · 2004-10-30 10:45 · Score: 4, Informative

Been doing this with 5 Maxtor Firewire 250gig drives for a good while, and regular ide drives for years before that. It's always been very stable and has had no problems with drives going bad as long as you replaced them quickly. I moved to firewire though, because it was much easier to see which drive went bad out of the set, and you could hot swap them.

Re:Works great by doublebackslash · 2004-10-30 10:55 · Score: 1

I agree, haven't had any probs, save for 1 drive crash, and i have migrated the array twice to new boxes.
I love the idea of firewire, too, it makes perfect sense, 'cause if you are gonna have raid reliability, then you might as well have hot-swap. (note to self: save up for firewire enclosures).

I would have 2 years of uptime, but NOOOOO 1 national power outage and 1 drive crash (perfect recovery). Uptime 71 days =( [I WAS at 260 at one point]
just make sure that the drives are up to that much spin-time, and mix brands so that 2 crashes at once are less than likely. Once you can afford it get a hot spare or two for backups for when one does fail.

--
md5sum /boot/vmlinuz
d41d8cd98f00b204e9800998ecf8427e /boot/vmlinuz
Re:Works great by Gherald · 2004-10-30 10:56 · Score: 1

You can hot swap SATA. Deffiately the way to go nowdays, seeing as they are only $1-5 more expensive than their IDE counterparts.

--
The unofficial /. digest
Re:Works great by dnoyeb · 2004-10-30 10:59 · Score: 1

Are you making one big raid partition on each drive and is you booting from this raid partition?

Or do you have still a seperate boot partition, and then the raid partition is seperate and maybe even another seperate swap partition?

I am hoping to do raid but it would be MUCH nicer if all i had was 1 big partition. Or more specifically, if using RAID did not force me into a custom partitioning scheme. Possible?
Re:Works great by Anonymous Coward · 2004-10-30 11:02 · Score: 0

Problem with SATA: the shielding of the connectors isn't specified at all in the standard. I.e. not really for drives not in your case. I.e. hot swapping gets harder again.
Re:Works great by k.ellsworth · 2004-10-30 11:08 · Score: 5, Interesting

Normally a drive crash anonunces itself some time before... use the smartctl tool.
that tool checks the SMART info on the disk about posible failures..

I do a lot of software raids and with smartctl, no drive crash has ever surprised me. i always had the time to get a spare disc and replace it on the array before something unfunny happened.

do a smartctl -t short /dev/hda every week and a -t long every month or so ...

read the online page of it:
http://smartmontools.sourceforge.net/

A example of a failing disc:
http://smartmontools.sourceforge.net/exampl es/MAXT OR-10.txt
a example of the same type of disc but with no errors:
http://smartmontools.sourceforge.net/exam ples/MAXT OR-0.txt

Software raid works perfect on linux... and combined with LVM the things gets even better

--
Putting a windows cd backwards, plays evil messages, but it gets worse, putting it right, installs windows.
Re:Works great by Gherald · 2004-10-30 11:12 · Score: 2, Insightful

SATA is meant to be used internally, yes.

OP said he switched to fireware for hot swapping reasons alone, that is why I mentioned SATA as an alternative.

If you're beant on having an external RAID 5, you're probably safest going with a DIY gigabit ethernet NAS.

--
The unofficial /. digest
Re:Works great by AIX-Hood · 2004-10-30 11:49 · Score: 1

Well, you can't do it via software raid because all of each drive is consumed for the raid. I have a small machine with a smallish hard drive for boot, and then all the firewire ones hanging off it outside on the desk. You'd also need a machine that has a bios capable of botting of firewire which some news ones do.
Re:Works great by RandomJoe · 2004-10-30 11:50 · Score: 1

You could do one big partition, but I made three. I didn't want the system partition to be filled up if I accidentally recorded too much with MythTV or something, and the drives I happened to wind up with made a better fit this way too.

I have two 160G, and two 120G drives. The system partition is a RAID 1 set, 30G each on the two 160s, along with a RAID 1 set for swap (1G, thereabouts). I then made four 120G partitions and made them the RAID 5 set for data.

I had actually considered not doing the mirroring for the system and swap, but especially when the drive sizes worked out this way, it seemed silly not to protect that too.
Re:Works great by dossen · 2004-10-30 13:21 · Score: 1

Well, if you are using the linux md raid system, it can use partitions, so if you wanted you could have a small boot- and/or root-partition mirrored on all of the drives with the rest as raid5. Not saying it is the way to go, but it is possible.
Re:Works great by glamslam · 2004-10-30 17:25 · Score: 1

Why not use USB instead of firewire? Isn't USB2 faster?
Re:Works great by AIX-Hood · 2004-10-30 17:35 · Score: 1

Well it's only faster by a little bit and I'm running the thing off a 600mhz P3 with a firewire card in there. This was a good while before USB2 was available. Since most external drives these days do both standards, you could switch between them whenever you needed. It all depends on what kind of linux driver implementation you can get your hands on.
Re:Works great by amorsen · 2004-10-30 20:40 · Score: 1

USB2 has higher theoretical bandwidth (480Mbps vs 400Mbps of firewire-400). In practice firewire is faster, particularly under Linux. Also, there's firewire-800.

--
Finally! A year of moderation! Ready for 2019?
Re:Works great by Jennifer+E.+Elaan · 2004-10-31 03:17 · Score: 1

Hotswapping SATA drives is easy, if the OS supports it (Linux, regrettably, does not yet). Just get hotswap drive enclosures. They fit in a 5 1/4" bay, and make drive swapping quite simple.

--
Hardware, software, and blinking lights!
Re:Works great by Carnildo · 2004-11-01 10:36 · Score: 1

The only drive failure I've ever had was one that wasn't predicted by SMART: the drive spontaneously lost the ability to find track 0, and consequently, the whole drive was unreadable until the controller electronics were replaced -- not a cheap operation.

--
"They redundantly repeated themselves over and over again incessantly without end ad infinitum" -- ibid.

Stick with hardware RAID by chrispyman · 2004-10-30 10:46 · Score: 2, Informative

Generally for situations where you really need to make sure the data stays safe, I'd just stick with hardware. If you can spend that much on some harddrives, I don't see why you can't spend the money on hardware.

Though from what I hear, software RAID on Linux works decently.

Re:Stick with hardware RAID by Anonymous Coward · 2004-10-30 10:50 · Score: 5, Insightful

Actually, a big disadvantage to hardware RAID is what happens if your controller fails.

Consider--your ATA RAID controller dies three years down the road. What if the manufacturer no longer makes it?

Suddenly, you've got nearly 2 TB of data that is completely unreadable by normal controllers, and you can't replace the broken one! Oops!

Software RAID under Linux provides a distinct advantage, because it will always work with regular off-the-shelf hardware. A dead ATA controller can be replaced with any other ATA controller, or the drives can be taken out entirely and put in ANY other computer.
Re:Stick with hardware RAID by dnoyeb · 2004-10-30 10:57 · Score: 1

Is hardware supposed to be better? If so, why?

From what I read software is just as good as hardware RAID these days, and sometimes better. But its only what I read, i dont have first hand info.
Re:Stick with hardware RAID by kgasso · 2004-10-30 10:58 · Score: 1, Informative

> Generally for situations where you really need to make sure
> the data stays safe, I'd just stick with hardware. If you can
> spend that much on some harddrives, I don't see why you can't
> spend the money on hardware.

Truer words were never spoken. I don't know the status of the more recent software RAID implmentation in Linux, but I do know that bugs in the old one send 2 arrays in 2 different mission critical servers of ours down in a hailstorm of fire and brimstone.

We had one drive get booted from the array for having corrupted data, so the load on the other drives shot up a bit. We think that the increased load made the software RAID driver start lagging in writes to the disks, causing more corruption on another drive, until we were down to a steaming pile of rubble.

Happened 2 seperate times on 2 different machines, as well. We're sticking to hardware from now on.
Re:Stick with hardware RAID by mortonda · 2004-10-30 11:02 · Score: 2, Informative

RAID 5 hardware tends to be rather expensive, and most RAID hardware tends to be "pseudo hardware", the drivers for the raid card make the CPU do the actual work anyway. Your 500Mhz CPU is faster than all but the most expensive RAID controllers anyway.

Stick with Linux RAID. It knows how to do it better.
Re:Stick with hardware RAID by fleabag · 2004-10-30 11:04 · Score: 2, Interesting

I would support the sentiment.

Back when I was using a PII-450 as a file server, I tried out software RAID on 3 x 80 Gb IDE disks. It mostly worked fine - except when it didn't. Generally problems happened when the box was under heavy load - one of the disks would be marked bad, and a painful rebuild would ensue. Once two disks were marked bad - I follwed the terrifying instructions in the "RAID How-To", and got all my data back. That was the last straw for me...I decided that I didn't have time to watch rebuilds all night. Note that this may have been caused by my crummy Promise TX-100 cards, I never bothered to investigate.

I got an Adaptec 2400 IDE controller, and it hasn't blinked for two years. One drive failure, and the swap in worked fine.

If the data is important to you - go hardware. If you want to lean something, and have the time to play, then sofware is OK. Just run frequent backups! If the data is really important to you, buy two identical controllers, and keep one in the box for when the other craps out. Having a perfect raidset, with no controller to read them, would be annoying.
Re:Stick with hardware RAID by Anonymous Coward · 2004-10-30 11:04 · Score: 1, Insightful

make sure the data stays safe
Um, perhaps my understanding is wrong, but isn't RAID5 intended solely for reliability (that is, for making the storage system tolerant of a single drive failure, and thus increase its mean uptime). If you want the data to stay safe then use a backup, not a RAID.
In general (not replying you your otherwise quite correct post, please don't feel browbeaten) I really wonder
a) why anyone would need the additional uptime in an in-home setting and
b) what the point of a generic IDE raid5 is anyway. When one drive dies, the system keeps running with the hotspare. On a commercial array (or using hot-pluggable storage like firewire) you can pull out the bad drive, put in a new one, and the system rebuilds that as the hotspare, all without any loss of service. But with regular ATA (and I guess SATA, although I'm not so sure) you can't hotswap, so you have to powerdown the array to swap in the new drive - at which point the reliability you got from RAID5 is gone. Hmm, well, I suppose it's less downtime than you'd have restoring from backups, but it's questionable if that's worth the ongoing performance hit the RAID5 (even a hardware one) would cause.
Re:Stick with hardware RAID by Anonymous Coward · 2004-10-30 11:05 · Score: 0

-1 redundant my dear!
Re:Stick with hardware RAID by Anonymous Coward · 2004-10-30 11:05 · Score: 0

Not true. Of course it is standardized how RAID stores the data on individual drives. I.e. you can simply replace one controller with any other one which supports the same RAID level.
Re:Stick with hardware RAID by kcbrown · 2004-10-30 11:14 · Score: 5, Informative

Generally for situations where you really need to make sure the data stays safe, I'd just stick with hardware. If you can spend that much on some harddrives, I don't see why you can't spend the money on hardware.

I disagree with this. Here's why: the most important thing is your data. Hardware RAID works fine until the controller dies. Once that happens, you must replace it with the same type of controller, or your data is basically gone, because each manufacturer uses its own proprietary way of storing the RAID metadata.
Software RAID doesn't have that problem. If a controller dies, you can buy a completely different one and it just won't matter: the data on your disk is at this point just blocks that are addressable with a new controller in the same way that they were before.
Another advantage is that software RAID allows you to use any kind of disk as a RAID element. If you can put a partition on it, you can use it (as long as the partition meets the size constraints). So you can build a RAID set out of, e.g., a standard IDE drive and a serial ATA drive. The kernel doesn't care -- it's just a block device as far as it's concerned. The end result is that you can spread the risk of failure not just across drives but across controllers as well.
That kind of flexibility simply doesn't exist in hardware RAID. In my opinion, it's worth a lot.
That said, hardware RAID does have its advantages -- good implementations offload some of the computing burden from the CPU, and really good ones will deal with hotswapping disks automatically. But keep in mind that dynamic configuration of the hardware RAID device (operations such as telling it what to do with the disk you just swapped into it) is something that has to be supported by the operating system driver itself and a set of utilities designed to work specifically with that driver. Otherwise you have to take the entire system down in order to do such reconfiguration (most hardware RAID cards have a BIOS utility for such things).
Oh, one other advantage in favor of software RAID: it allows you to take advantage of Moore's Law much more easily. Replace the motherboard/CPU in your system and suddenly your RAID can be faster. Whether it is or not depends on whether or not your previous rig was capable of saturating the disks. With hardware RAID, if the controller isn't capable of saturating the disks out of the box, then you'll never get the maximum performance possible out of the disks you connect to it, even if you have the fastest motherboard/CPU combination on the planet.

--
Use 'slashdot stuff' in the subject line in any email you send me if you want to get past the spam filter.
Re:Stick with hardware RAID by k.ellsworth · 2004-10-30 11:18 · Score: 2, Informative

true... if your are not getting a seriuos RAID controller forget about hardware raid... is just the same of linux software raid but made by the controller bios and the controller driver.... that's why many "pseudo hardware" controllers make the array but booting into linux, linux only see's the harddisks and no array ... becuase there is no real array...

for the trolls: a real raid controller for me is a HP/Compaq smartarray, IBM server raid, Intel RAID, some megaraid controller...

i have a Proliant DL380g2 at my home, and it has a smartarray 5i with 32MB RAM of it's own, and a risc CPU for the array computing...

doing any raid configuration on the machine... the system CPUs are not affected.

--
Putting a windows cd backwards, plays evil messages, but it gets worse, putting it right, installs windows.
Re:Stick with hardware RAID by aaronl · 2004-10-30 11:20 · Score: 1

Actually, RAID5 is faster than single disc. On writes you have a hit on whatever processor is doing your parity calculations. On reads, you get access to the data at faster than single spindle rates and don't need to use parity (but your implementation may). If you have CPU to spare, your write performance improves over single drive as well. You get the performance benefits of RAID0 on reads, and slightly slower than RAID0 write rates.
Re:Stick with hardware RAID by ebunga · 2004-10-30 11:27 · Score: 2, Insightful

That's why you do this thing called "HAVE A SPARE ON HAND." Sure, it costs more money, but you *are* going for the highest reliability aren't you?
Re:Stick with hardware RAID by ericdano · 2004-10-30 11:32 · Score: 1

It's called EBAY. Try it, you can find everything there.........

--
It's either on the beat or off the beat, it's that easy.
I moderate therefore I rule!
--
Re:Stick with hardware RAID by ericdano · 2004-10-30 11:34 · Score: 1

I agree, I have a 2400 Adaptec. It works great. Rebuild times were slow with a Dual 450Mhz machine (rebuilding a 500Gig array), but I could live with it. The Adaptec controller works great under various other OSes as well (FreeBSD, Windows).

--
It's either on the beat or off the beat, it's that easy.
I moderate therefore I rule!
--
Re:Stick with hardware RAID by kcbrown · 2004-10-30 11:42 · Score: 2, Interesting

In general (not replying you your otherwise quite correct post, please don't feel browbeaten) I really wonder a) why anyone would need the additional uptime in an in-home setting

The uptime isn't the reason for using RAID at home. Data integrity is.
With RAID, I don't lose all my data (or, if I take regular backups, all the data since the last backup) in the event that a drive fails, as long as I . A good RAID-5 setup will give me better read speeds than a single disk, at the cost of some write speed. Since reads are generally much more common than writes on a home system, this is an overall win.
However, these days disks are big enough that a RAID 0 configuration is reasonable, and that's what I have now. I get better write speeds and similar read speeds.
In any case, backups are no substitute for a good RAID setup. In fact, I would argue that the home situation is much more appropriate for RAID, because there simply is no good backup solution for home use -- hard disks are orders of magnitude larger than any reasonably-priced backup medium you can find. Only businesses can afford the kind of backup solutions that are capable of backing up the amount of data that's typical on a home system today without burning through a bunch of backup media.
and b) what the point of a generic IDE raid5 is anyway. When one drive dies, the system keeps running with the hotspare. On a commercial array (or using hot-pluggable storage like firewire) you can pull out the bad drive, put in a new one, and the system rebuilds that as the hotspare, all without any loss of service. But with regular ATA (and I guess SATA, although I'm not so sure) you can't hotswap, so you have to powerdown the array to swap in the new drive - at which point the reliability you got from RAID5 is gone. Hmm, well, I suppose it's less downtime than you'd have restoring from backups, but it's questionable if that's worth the ongoing performance hit the RAID5 (even a hardware one) would cause.

Downtime isn't an issue for home use anyway. But loss of data is. That's why RAID solutions without hotswap capability are perfectly adequate for home use.

--
Use 'slashdot stuff' in the subject line in any email you send me if you want to get past the spam filter.
Re:Stick with hardware RAID by kcbrown · 2004-10-30 11:54 · Score: 1

Sigh...
I wrote:
With RAID, I don't lose all my data (or, if I take regular backups, all the data since the last backup) in the event that a drive fails, as long as I ...

... replace the dead drive in time.

--
Use 'slashdot stuff' in the subject line in any email you send me if you want to get past the spam filter.
Re:Stick with hardware RAID by ErikZ · 2004-10-30 11:58 · Score: 1

How the heck am I supposed to backup 500 GB of data? Set up another raid and copy it over?

--
Democrats or Republicans. They are both taking us to the same place and they are not afraid of us anymore.
Re:Stick with hardware RAID by Anonymous Coward · 2004-10-30 12:06 · Score: 0

A good tape (DVD, CDR, etc.) backup regimen for your data is all that is required to deal with this issue.
Re:Stick with hardware RAID by pjrc · 2004-10-30 12:10 · Score: 5, Interesting

Consider--your ATA RAID controller dies three years down the road. What if the manufacturer no longer makes it?
This happened to me. The card was sorta still working... could read, with lots of errors usually recoverable, but writing was flakey.
Luckily, even after about 3 years, 3ware (now AAMC) was willing to send me a free replacement card. They answered the phone quickly (no long wait on hold), they guy I talked with knew the products well, and he had me email some log files. He looked at them for about a minute, asked some questions about the cables I was using, and then gave me an RMA number.
The new card came, and my heart sank when I saw it was a newer model. But I plugged the old drives in, and it automatically recognized their format and everything worked as it should.
This might not work on those cheapo cards like Promise that really are just multiple IDE controllers and a bios that does all the raid in software. Yeah, I know they're cheaper, but the 3ware cards really are very good and worth the money if you can afford them.

--
PJRC: Electronic Projects, 8051 Microcontroller Tools
Re:Stick with hardware RAID by Fweeky · 2004-10-30 12:57 · Score: 1

"isn't RAID5 intended solely for reliability"

No. RAID-5 has advantages in performance too, mainly in the ability to service multiple accesses. Of course, if you need a lot of storage, and hence a lot of disks, performance isn't that much of a concern; it's more down to the massively increased probability of one or more of your drives experiencing early failure.

"If you want the data to stay safe then use a backup, not a RAID."

Another RAID array can make a good backup solution, and with relatively unimportant data, even a single one can reduce the probability of failure to a level where not backing up is acceptable. Say you want to keep your DVD/music collection on live storage -- you already have backups, but restoring is a big job...

"why anyone would need the additional uptime in an in-home setting"

I dunno about you, but my time's pretty precious. If I can avoid spending a few hours rebuilding a drive, or my entire OS every year or so, even a moderately expensive setup will more than pay for itself.

As for hotswap; SATA supports it natively, and there are plenty of cases about which include hotswap drive bays. You can even get racks/caddies which fit in 5.25" bays. They're a bit pricy, but then so are big monitors and powerful graphics cards and top end CPU's; so what?
Re:Stick with hardware RAID by Anonymous Coward · 2004-10-30 13:25 · Score: 0

But with regular ATA (and I guess SATA, although I'm not so sure) you can't hotswap, so you have to powerdown the array to swap in the new drive - at which point the reliability you got from RAID5 is gone.

In addition to the other points that were made, scheduled downtime is a lot less of a pain than unscheduled downtime. Depending on what you're doing, you can most likely transfer service seemlessly to a temporary system, and then transfer back after you're done with the fix.
Re:Stick with hardware RAID by IgnorantSavage · 2004-10-30 13:52 · Score: 1

Regarding: "The uptime isn't the reason for using RAID at home. Data integrity is."

and

"However, these days disks are big enough that a RAID 0 configuration is reasonable, and that's what I have now"

Perhaps you meant RAID 1? I agree that mirroring is often enough (I run this for my main PC drive), unless you want really large amounts of redundant storage (> 250GB or so with current drive prices).

If you really mean RAID 0, you need to know that it is not redundant and you will lose all of your data if any drive fails. RAID 0 is mainly a performance configuration. It can be combined with redundant RAID types (i.e. RAID 0+1/0+5) to get redundancy.

Perhaps I am misinterpreting your post, but it sounds like you may have a serious misconception that could lead to a real problem if any drive fails.

Note also that even if you have to power down to swap a RAID drive that fails, you still have the reliability. You just don't have the uptime, which is not important for many home users but critical for many companies.
Re:Stick with hardware RAID by bltfast32 · 2004-10-30 14:01 · Score: 1

I'd fully disagree here. I've used it in 0 and 1 configurations and it worked perfectly.

Failures were easy to manage and rebuilds were a breeze. Managed 5 or so failures with no issues.

It's all well documented and quite simple once you get the hang of it.
Re:Stick with hardware RAID by Jeff+DeMaagd · 2004-10-30 14:07 · Score: 1

What I find is that often old RAID cards are available pretty cheaply on eBay. I get most of my computer hardware via eBay and I've done pretty well.
Re:Stick with hardware RAID by dougmc · 2004-10-30 14:10 · Score: 1

Set up another raid and copy it over?
That's one way. Of course, it doesn't have to be a RAID at all -- just 500 GB of disk. Could be on a different system (in fact, that's probably safest.) If your data compresses well, and you can compress it as you copy it over, you may not even need the full 500 GB.
Of course, other options including spending oodles on a DLT drive and media -- probably a good deal more than you spent on the hard drives in the first place. Or a DVD-R drive and 120 disks. (I know, it sounds crazy, but it works for some people.)
Or maybe you can tolerate losing most of that 500 GB of data (if it's your porn collection, you may not really care that much.) In that case, you back up what you need to another disk or to some other backup solution, and leave the rest to chance.
In any event, for important data, RAID 5 is not a subsitute for backups. It may protect you against the loss of one disk (if you replace it before another fails and rebuild the array) but it will do NOTHING to protect you against `rm -rf *' or the entire machine sh*tting itself and taking your data with it (which happens more than we'd like, RAID or not, though RAID seems to make it happen more often, on pretty much any OS.)
Re:Stick with hardware RAID by sjames · 2004-10-30 14:12 · Score: 1

We had one drive get booted from the array for having corrupted data, so the load on the other drives shot up a bit. We think that the increased load made the software RAID driver start lagging in writes to the disks, causing more corruption on another drive, until we were down to a steaming pile of rubble.

That seems unlikely. The load on the unaffected disks won't change at all when a disk fails(In any RAID). If the load on the drive caused corruption, they were crap from the start.

More likely is that you had several disks all the same age. They reached the far side of the bathtub curve and failed.

I prefer to use mixed age disks in a RAID. It doesn't guarantee anything, but it does make it much more likely that disk failures will be convieniantly widely spaced out.

I've encountered several hardware raids that had to be written off after a controller failure. I've never had to write a Linux softraid off.

Of course, any RAID can fail beyond repair. RAID reduces the odds of it, but never eliminates the possability.My favorite case of that was a nice 10 disk hardware raid system. The power supply failed catestrophically and as a result, 208V was bridged to the SCSI. Naturally, the data was gone.
Re:Stick with hardware RAID by sjames · 2004-10-30 14:15 · Score: 1

Your 500Mhz CPU is faster than all but the most expensive RAID controllers anyway.

For that matter, the 500MHz CPU (and MB) is cheaper than the RAID card.
Re:Stick with hardware RAID by kcbrown · 2004-10-30 14:26 · Score: 1

Perhaps you meant RAID 1?

Yeah. Sigh. Guess this is one of those days... :-(

--
Use 'slashdot stuff' in the subject line in any email you send me if you want to get past the spam filter.
Re:Stick with hardware RAID by WindBourne · 2004-10-30 15:15 · Score: 1

Sure, it costs more money, but you *are* going for the highest reliability aren't you?
If going for the highest reliability, then you would be doing software raid, plain and simple.

--
I prefer the "u" in honour as it seems to be missing these days.
Re:Stick with hardware RAID by CliffEmAll · 2004-10-30 16:23 · Score: 1

I am certainly no expert on the matter, but I have had software RAID5 running on a FC1 installation for several months with no problems. Sure, if you are running a business and the information on this machine is mission-critical I would think hardware would be much more reliable and efficient, but that doesn't seem to be the focus of this question.
For a poor college student like myself, cobbling together software RAID from commodity hardware is relatively inexpensive and easy. Given that a) I don't need to store huge amounts of data and b) I was building the system to be fault-tolerant, I decided I could just buy cheap used hard drives from eBay. For connecting all the drives, I bought an old Promise Ultra66 IDE controller for $15 at a Marketpro show.
It isn't remotely fast, but it has solidly protected my personal PostgreSQL databases and music collection. It should go without saying, but YMMV. Good luck!
BTW, I haven't seen anyone else mention this so I will. Buying a lot of identical drives from a manufacturer may not be the best idea. All the drives you purchased were probably manufactured in the same batch, and they might be likely to fail together.
Re:Stick with hardware RAID by Anonymous Coward · 2004-10-30 16:31 · Score: 0

I have used both raid5 in software (5x 47G ST446452W SCSI disks) and raid5 in hardware (3ware 6xxx, 7xxx cards, various ide disks). I have to say that the systems that gave me the least grief were the 3ware cards.

The software raid seemed to drop one disk off the raid group if someone sneezed in the server room. On the software raid, it very much does work, however the system I have, based on redhat8 (and upgraded raid tools) requires much more hand holding over the long haul.

On other systems I manage, I have 3ware controllers, 4 and 8 port versions. These work quite well, have reasonable tools to support them, and are fast. (very fast) The raid rebuild managed by the controller card is much faster as it doesn't require the CPU time to manage the disks. The 3ware card also comes with some software for web browser administration which works well in my situation.

I can also say that 3ware tech support was quite good compared to others I have had the pleasure of calling. Their hold time was short, the tech understood the products, sent me a diagnostic tool via email. Upon reading the results on the screen, he gave me an RMA number. The card was actually one of the first ones they ever shipped, mod-wire and all. The replacement was a much later version, half the size and twice the cache, however it worked fine with the disk formats I had installed on the previous card. I am also pleased that they make their driver available in source format, and is part of the linux kernel distribution.

If I knew then what I know now, I would have spent the extra money to buy a 3ware card.

I do not have a personal interest in 3ware other than I like using their products in many servers I build.
Re:Stick with hardware RAID by realdpk · 2004-10-30 16:43 · Score: 2, Interesting

Yep. Maybe even set up a pseudo-"RAID5" with 3 RAID5 servers. :) Costs _WAY_ less than tape, and is far more reliable. (I hate tape, can you tell?)
Re:Stick with hardware RAID by bergeron76 · 2004-10-30 16:43 · Score: 1

I have to second this second hand. I have never personally used a 3ware card (however, it's on my wish-list; I already have the drives). I did a ton of research and I've heard nothing but great things about the 3ware cards.

I aspire to have a PCI one eventually.

Anyway, the point is that if hardware is an option for you, I don't think you can go wrong with 3ware from what I've heard (and researched).

--
Don't think that a small group of dedicated individuals can't change the world. It's the only thing that ever has.
Re:Stick with hardware RAID by ti.payn · 2004-10-30 17:21 · Score: 1

A man of my own mind ... I have used a 3 RAID 5 setup with good results: One array for use, one for on-site backup and one off-site for backup replication. You are correct re: tape reliability and furthermore you dont' have to pay someone (daily) to fuck around with the tapes. Of couse, everytime RAID backup is mentioned, some fuddy duddy 1960's Admin get's into a hissy fit screaming "RAID isn't a backup solution" but if you can get past that you can see the awesome power of using multiple RAID arrays as a backup solution.
Re:Stick with hardware RAID by mauriceh · 2004-10-30 18:00 · Score: 1

Software RAID usually outperforms most hardware RAID controllers by a factor of 2 or 3 to one in RAID5.
Software RAID allows a lot more options and choices.
And the big one:
Software RAID allows one to use a common metadata structure that is not proprietary.
So one may build a RAID set on one controller, and easily move it to a different one.

--
Maurice W. Hilarius Voice: (778) 347-9907
Re:Stick with hardware RAID by XNormal · 2004-10-30 18:03 · Score: 2, Informative

Suddenly, you've got nearly 2 TB of data that is completely unreadable by normal controllers, and you can't replace the broken one! Oops!

This is also a good reason to use mirroring rather than fancier schemes like striping or RAID-5, if you can afford the capacity hit. You can always mount the drive individually.

--
Stop worrying about the risks of nuclear power and start worrying about the risks of not using nuclear power.
Re:Stick with hardware RAID by JayAEU · 2004-10-30 18:08 · Score: 1

What most people fail to realize is that having a RAID-subsytem for storage doesn't relieve you from organizing a proper backup scheme.

Any RAID, be it software or hardware, can only protect an individual failure of harddisks, but not against corrupt filesystems or users deleting files deliberately.
Re:Stick with hardware RAID by 1lus10n · 2004-10-30 19:04 · Score: 1

hardware raid is faster since the work is offloaded to the card, this also means you can run it on a slow system and not have issues with latency/load.

--
"Two things are infinite: the universe and human stupidity; and I'm not sure about the the universe." --Albert Einstein
Re:Stick with hardware RAID by steveha · 2004-10-30 19:12 · Score: 1

I have been doing software RAID for years now, and I'm pleased with it. Even my K6-III/450 does a decent job with two 30 GB drives in a RAID 1; rebuilds take less than 40 minutes.

With a very inexpensive server built from an EPIA-M motherboard (VIA C3 processor, 1 GHz) I have a RAID 5 with three 120 GB drives. It rebuilds in about half an hour.

My latest server, also an EPIA-M with a 1 GHz C3, has a pair of Western Digital 40 GB hard drives in a RAID 1 mirror. I haven't had to do a full rebuild yet, but based on what it's been doing as I have been setting it up, I'd estimate a full rebuild as being under 20 minutes.

I have never had any trouble with my software RAID; it just works.

steveha

--
lf(1): it's like ls(1) but sorts filenames by extension, tersely
Re:Stick with hardware RAID by Anonymous Coward · 2004-10-30 20:37 · Score: 0

Mirroring only kinda limits you by hard drive size, if you want to be able yto recover without the controller.
Re:Stick with hardware RAID by Donny+Smith · 2004-10-30 20:41 · Score: 1

>Consider--your ATA RAID controller dies three years down the road. What if the manufacturer no longer makes it?

Buy a different controller or use software RAID to create a new RAID.

Then restore your backup data.
Re:Stick with hardware RAID by amorsen · 2004-10-30 20:46 · Score: 1

Hardware raid is slower since the cards have damn slow CPUs on them. And yes, most of them use regular CPUs. The best RAID accelerators you can buy are sold by AMD and Intel. They have names like "Xeon" and "Opteron".

--
Finally! A year of moderation! Ready for 2019?
Re:Stick with hardware RAID by Cheetahfeathers · 2004-10-30 20:52 · Score: 1

I want the May 21st version of resume.doc from my home directory. I've overwritten it 3 times since then, and then it was
corrupted. Oh wait.. you can't get it, you only store the current corrupt version, but 3 times. Back to tapes.
Re:Stick with hardware RAID by amorsen · 2004-10-30 20:54 · Score: 1

The 5i is dead slow for small writes on RAID-5. On the order of 5 writes/sec. Most often the OS caching papers over the problem, but if you do small synchronous writes you get performance breakdown. Hardware RAID controllers with non-volatile memory are supposed to shine at small writes (just write to memory, wait till you have a multimegabyte batch, push the batch to the disks.)
Anyway, unless you're doing video or you're low on disk capacity, go with RAID-10. And if you're willing to trade performance for disk capacity, you shouldn't be using SCSI RAID in the first place.

--
Finally! A year of moderation! Ready for 2019?
Re:Stick with hardware RAID by julesh · 2004-10-30 21:07 · Score: 1

Of course, 3 years is nothing. If it were a ten year old board they had been able to replace just like that, then I'd be impressed.
Re:Stick with hardware RAID by julesh · 2004-10-30 21:18 · Score: 1

Generally problems happened when the box was under heavy load - one of the disks would be marked bad, and a painful rebuild would ensue.

Were you running kernel 2.4.20? I ran this kernel for several months, and got corruption in a reiserfs file system and a mysql data file. Turns out there was a bug that sometimes caused incorrect data to be written when under heavy load in that kernel version. Note that this is completely independent of RAID usage, and might even have been _worse_ with hardware RAID because the same corrupted data would have been written to all volumes, not just one.
Re:Stick with hardware RAID by julesh · 2004-10-30 21:21 · Score: 1

Of course, the fact is that what you're doing isn't RAID. RAID isn't a backup solution -- if somebody nicks your computer or it catches fire or something you'll lose all of your data. What you're doing is offsite data replication, which is much better.

I use a combination of offsite replication and periodic backups to DVD-R, with occasional deltas being stored on CDRW until they get too large, at which point a new DVD-R is written.
Re:Stick with hardware RAID by Fallen_Knight · 2004-10-30 21:44 · Score: 1

3ware seems like a great company, i e-mailed them with a question about NCQ and got a decent answer back in a single day!

promise... 1 week and still no answer.
Re:Stick with hardware RAID by Anonymous Coward · 2004-10-31 00:17 · Score: 0

ridiculous, more than a 5 year warranty is way too impractical
Re:Stick with hardware RAID by julesh · 2004-10-31 01:11 · Score: 1

I don't care if you have to pay for the replacement, that isn't the point. The point is that after 10 years they probably won't be able to replace it with a compatible board.
Re:Stick with hardware RAID by JayAndSilentBob · 2004-10-31 01:18 · Score: 1

A couple of generic 160 gig drives from pricewatch ($70 each), combined with a few cheap enclosures from pricewatch ($20 each) and a safe deposit box across town ($30/year) ensures that even in the event of my home being completely wiped from the map, my data will survive. I'm running a 4 disk 160gb doftware raid-5 which has no troubles as of yet. Two disks are year old Maxtor disks and 2 of them are brand new generic pricewatch disks. I've had no problems or trouble with any of them. Before I could afford more disks, the 2 maxtors were running mirrored in software with no trouble for about a year. After I filled them up, I decided to triple my capacity and go with the raid-5. It's been wonderful. I set it up using the guide at http://www.tldp.org/HOWTO/Software-RAID-HOWTO-5.ht ml. I don't have any "hot" spares, but I have perfectly suitable replacement disks available to drop in once a primary drive fails. AND I have an offiste backup.

--

Love,
Jay and Silent Bob
Re:Stick with hardware RAID by new500 · 2004-10-31 03:13 · Score: 1

. . .

Actually, a big disadvantage to hardware RAID is what happens if your controller fails.

I'll bite :

1. In any MegaRAID device, I'll just swap the battery backed cache unit to another controller. Downtime hardly worth bothering about.

2. Isn't it commercial suicide - generally accepted idiomatically, anyhow - failing to support any "professional" level hardware long after RTM?

3. If you're interested in performance as well as redundancy, surely you'll want to offload all that work to a dedicated controller?

4. Isn't the "I" in RAID part of a compound noun : "inexpensivedisks" rather than globally reflexive?

5. Being conservative myself, forgive my bias, but would you actually specify your storage ot run on a Johnny - come - lately manufacturer's latest product rather than a company you've known to go the distance?

6. After my skepticism, you raise an interesting point - how many OTS RAID controllers support hardware redundancy?

7. That said, I have some controllers just chugging away year after year. I expect they are better built, and were more expensive originally than most enthusiast machines with an ad - hoc ATA RAID.

8. If you think good harware RAID is prohibitively expensive, tell yourself otherwise at www.scsi4me.com. That's a shameless plug, but i have no connection with them other than being a happy long standing customer.

== Idle Random Thoughts, Usual Disclaimers Apply ==
Re:Stick with hardware RAID by Jennifer+E.+Elaan · 2004-10-31 03:26 · Score: 1

Actually, from what I understand, many hardware RAID controllers have notoriously bad records for data safety. My partner, who works in a datacenter, has described one particularly bad controller that has a nasty habit of erasing all data and reinitializing the drives if anything unexpected happens. It can be something relatively small, shuffling the drives, or mismatching drives of different volumes. This isn't just a case of "the array doesn't come up". It will actually ERASE ALL DATA on the drives.
From what I understand, this isn't exactly uncommon behavior for the high-end hardware RAID controllers. Linux software RAID is much more permissive, even recognizing a drive shuffling and bringing up the RAID safely. We thoroughly stress-tested the array before deploying it, and we could not make it break.

--
Hardware, software, and blinking lights!
Re:Stick with hardware RAID by new500 · 2004-10-31 03:27 · Score: 1

a real raid controller for me is a HP/Compaq smartarray, IBM server raid, Intel RAID, some megaraid controller..

Please tell that to Dell UK. Just spent 4 wasted days speccing new workstations, and the time waster was waiting for their manufacturing to confirm that "PERCs" are now supplied by Adaptec. So for a U320 SCSI dual channel, you get 3rd rate hardware instead of robust LSI MegaRAID models they used to carry.

I have it on reasonable authority that this does not currently affect the USA, and may not apply to servers, but it still put a hole in my planning. How did I smell the rat? Well, the quote came in way too low, even for Dell being agressive against my comparative quote process. The only thing I advise anyone to be wary of regarding Dell is their sales process - it always seems too hurried, too thin on real specs and heavy on generic terms. That said, I made good friends with a rep who turned out (in the end) to be quite pleased he'd just learnt (as a function of my pleading corrections) a bunch about RAID which is 1. helping him get bigger sales now 2. got me a better discount. Hmm, strange that education can have immediate cash benefits, but it worked for me. In the end.
Re:Stick with hardware RAID by danpritts · 2004-10-31 03:35 · Score: 1

You're certainly right on the cost of tape but you have either been using crappy tapes and/or drives or you are sniffing glue if you think tape's not reliable.
Re:Stick with hardware RAID by new500 · 2004-10-31 03:37 · Score: 1

Your 500Mhz CPU is faster than all but the most expensive RAID controllers anyway.

I beg to differ, based on the anecdotalism that specialised hardware tends IME to ourperform generic hardware, and also this :

http://www.intel.com/design/iio/80321.htm

which is the controller embedded in the latest LSI boards.

Specs are as follows, which don't seem lazy to me :

400, 600 MHz Intel XScale® Core High-performance with low power

133 MHz PCI-X Interface

200 MHz DDR SDRAM Bus

Up to 1GB memory support

1.6GB/s Internal Bus

32-bit Local Bus
Re:Stick with hardware RAID by sjames · 2004-10-31 03:38 · Score: 1

Any RAID, be it software or hardware, can only protect an individual failure of harddisks, but not against corrupt filesystems or users deleting files deliberately.

Fully agreed. Storage failure is only one of several ways to lose data. Unfortunatly, a lot of people seem to think RAID is a panacea.
Re:Stick with hardware RAID by Fweeky · 2004-10-31 04:14 · Score: 1

No, actually we have incremental backups going back several weeks. What, did you think tapes were the only storage devices with the magical property of supporting dump(8)?
Re:Stick with hardware RAID by sjames · 2004-10-31 05:25 · Score: 1

I beg to differ, based on the anecdotalism that specialised hardware tends IME to ourperform generic hardware, and also this

It is definatly NOT one of the more inexpensive controlers. It's not really faster than a 500MHz PIII, though it will run cooler with less power. I'll bet it can't be programmed to emailme if a disk goes bad.
Re:Stick with hardware RAID by Kyril · 2004-10-31 05:54 · Score: 2, Informative

Careful with that "always". There was a Compaq box using a RAID-1 controller that I couldn't immediately recover; I think it reserved some space at the front and put the partition table after that, so I couldn't readily mount it or even fdisk -l....
Re:Stick with hardware RAID by macdaddy · 2004-10-31 06:12 · Score: 1

Would you like to buy one? I bought a brand new 3Ware 8506-12 back in February and never put it into service. I haven't been able to find a case suitable for the massive array I want to build and the drive cages I want to use. The 3Ware box is still in its shrinkwrap. I have a 7506-4LP and a 7000-2 and love them. These things kick ass. I'll have to buy another 3Ware card if I ever find the right double-wide case but that's ok; I'd rather have the money back in the mean time. The lowest price Froogle has that card for now is $690. I'd part with it for $575. UPS shipping can't be too much. Are you interested? I'll probably put it up on eBay before long if no one I know wants to buy it. BTW, don't put Western Digital "special edition" drives on a RAID controller unless you talk WD out of the RAID firmware for their drives. I have 8 WD1200JB drives (8MB cache!) and they don't perform well in a RAID setup. However I have a number of Maxtor 7Y250P0 (and M0) and 6Y200P0 (and M0) drives that kick ass. Excellent buy if you're looking for good SATA drives. Are you interested in the controller? Post here if you are and we'll work something out.
Re:Stick with hardware RAID by macdaddy · 2004-10-31 06:15 · Score: 1

If you're running a production system with important data on a 10-year old array that has a single point of failure than you are reckless and irresponsible. Any sysadm worth his salt would have an identical controller and drives on standby. They would also replace that ancient system long before its MTBF was reached and age became a factor. Can they replace a 10-yr old controller? Who cares? It shouldn't ever come to that.
Re:Stick with hardware RAID by macdaddy · 2004-10-31 06:39 · Score: 1

I don't know what brand of cards you're used to using but all the decent hardware controllers I've used are many times faster. LSI's Megaraid, all the 3Ware controllers. Hell they were all faster. The only "hardware" RAID controllers I've ever seen run slower are the POS software/hardware controllers that Promise and High-Point make. Those are worse than Windows software RAID, infinitely slower than Linux software RAID. Offlaoing the RAID5 process to a decent hardware controller is always faster in my experience.
Re:Stick with hardware RAID by macdaddy · 2004-10-31 06:47 · Score: 1

The sad part is that there aren't any inexpensive and massive backup systems. I can buy 1TB of HD space for about $500. How much does a TB of a backup medium run plus the drive to handle it? $3000? That's the worst part about it. That makes people like me rely on RAID5 more than backups. The economics of a good backup system just don't exist.
Re:Stick with hardware RAID by ti.payn · 2004-10-31 08:26 · Score: 1

Yeah, I agree but you would be surprised how many people think that unless you are using tape, you are not doing backup. That's it: If it's not on tape, it's not backup (the idea that tape is that reliable amazes me -- it never is more reliable than when comparing it against disk based backup ... suddenly, people act like tape never corrupts). If someone is talking about one Raid 5 array and saying "I'm all set" then, yes, that is a shitty backup plan. But one set of tapes stored on-site has risks as well.
Re:Stick with hardware RAID by ti.payn · 2004-10-31 08:37 · Score: 1

Believe me, I am well aware of versioning. But that is mostly a software question: All you have to do is tell the software to see your array and then program it the way you want it (daily for two weeks, then monthly for a year, etc.). Then you can just sync off-site with another array. The arrays themselves are stupid - it's just data to them and they will synch whatever you have and to the software it's just a storage container. To me, tape has always felt like a pain in the ass (e.g. a pain in the ass that you have to pay a lot for, pay people to operate every day and does not - outside of some classroom discussion - have any increase in reliability that the extra money would result in). If a disk goes bad, I just put in another one and go about my business. I don't even have to put my sandwich down. Everything can be looked at via a web interface and thus set to send warnings (e.g. get an email that you need a new disk). Hey, if you want to use tape - fine. It's been done forever and the results are well known. I am just saying that there are other options that are seriously overlooked.
Re:Stick with hardware RAID by Ed+Random · 2004-10-31 09:16 · Score: 1

How the heck am I supposed to backup 500 GB of data? Set up another raid and copy it over?

Yup - that's exactly the setup I have at home:

- 4x 160GB SW-RAID5 on the master fileserver
- 4x 120GB SW-RAID0 on the backup fileserver

The 120GB disks used to make up the master fileserver; they now act as a safety net against complete data loss on the master.

I run rsync to mirror all data from the master to the backup fileserver; in unattended mode, no deletes are performed.

About once a week I explicitly check the RAID statuses and issue the 'purge' command on the backup fileserver. This protects me against accidental file deletes on the master fileserver.

--
-- Gxis! Ed.
Re:Stick with hardware RAID by sjames · 2004-10-31 10:51 · Score: 1

The economics of a good backup system just don't exist.

You're too right! I tend to over-rely on RAID at home as well. For critical data, I copy it to an extra RAID 1 that I leave unmounted most of the time.

The non-HD storage technology has fallen too far behind to be of much use in most situations.
Re:Stick with hardware RAID by macdaddy · 2004-10-31 11:14 · Score: 1

Yep. It's sad and very unfortunate. The more popular computers become and the more popular the digital medium becomes the more Mr and Mrs John Public will rely on their hard drives. It used to be that computers in the hands of average users contained their email (which wasn't essential to those types of users), some misc documents which they'll never remember anyhow, and maybe some Quicken data. The single biggest loss to the user was their bookmarks, of all things. Now with the prolifercation of things like digital cameras and digital camcorders our whole lives are stored on our hard drives. Not only do I have my basic bank account information on my computer but I'm now paying my bills from my computer. I'm buying and selling stocks from my computer. I'm tracking my IRA and cash investment accounts from my computer. I'm buying and installing software for which I have no physical installation medium, only a downloaded file. I'm buying books, music, even copies of my favorite movies on the Internet and downloading them to my hard drive. A hard drive failure now is an incredibly damning thing to any user. To an average user 8 years ago it was no biggie and they'd quickly forget about it.
A few years ago I could backup my hard drives with identical drives in a hot-swap chassis. It was actually quite economical. Now I'd have to have well over a dozen drives to pull off the same thing. An Exabyte, AIT or DDS autoloader for 1 coupe TBs is worth 4 or 5 times what my computer is worth and a dozen times what my car is worth. I sure hope something fills this niche soon (and that I buy their stock early) because we'll surely need it.
Re:Stick with hardware RAID by amorsen · 2004-10-31 11:33 · Score: 1

Compaq SA. IBM Serveraid. They benchmark beautifully, but real life performance is disappointing.

--
Finally! A year of moderation! Ready for 2019?
Re:Stick with hardware RAID by 1lus10n · 2004-10-31 13:42 · Score: 2

Then you dont know how to configure them or the software your using on top. What exactly is 'disappointing' performace ? And how are you benchmarking the cards or their performace ?

There are reasons that HW raid is used in all of the top end setups, because its faster. The CPU might not be 3Ghz Extreme edition or some other recognizable market friendly junk, but make no mistake the performance is there. Most of the time chips used on cards (not only RAID cards) are designed for that specific purpose and as such are much better at that job than a general "consumer" CPU. In the economy cards your usually dealing with a modified MIPS chip. Which is still more efficient and faster than software raid.

Don't compare a high end PC doing software RAID to a low end card. Apples to Apples. Of course a 5 year old serveraid card will suck compared to a new PC with SATA drives. Compare that same PC with a new high end card, the performance difference will be staggering. Especially if the PC is going to be used for normal tasks while doing software raid.

--
"Two things are infinite: the universe and human stupidity; and I'm not sure about the the universe." --Albert Einstein
Re:Stick with hardware RAID by amorsen · 2004-10-31 21:16 · Score: 1

The last time I dealt with RAID was Compaq stuff, up to and including the SA-4200. Disappointing performance was, say, fewer than 7 small synchronous writes per second to a RAID-5 volume. And fewer than 3 per second with the low-end cards.

--
Finally! A year of moderation! Ready for 2019?
Re:Stick with hardware RAID by Phreakiture · 2004-11-01 08:11 · Score: 1

Yep. Maybe even set up a pseudo-"RAID5" with 3 RAID5 servers. :) Costs _WAY_ less than tape, and is far more reliable. (I hate tape, can you tell?)

It is okay to hate tape. The bit where you and I disagree, though, is that redundancy is not backup, and vice-versa. Redundancy helps if you have a hardware failure, but will not help you if your file system gets corrupted (still possible with RAID) and will not help you if you hit enter after an 'rm' command and then go "Oh shit!"

That in mind, you might want to bolster a RAID array of any generation with either a manual mirror to a different device/array/computer or, even better, periodically do an incremental backup to a tape image (i.e. a TAR or CPIO file) on a different device/array/computer. That way, you are not just protected from boneheaded hardware design, but also from boneheaded software design and boneheaded users :-)

On another note, I once installed a server for a bank that was paranoid about data loss. The server had 13 9GB drives. Ten were arranged into two groups of 5, configured as RAID5, and then mirrored between the two groups.

Another logical volume on that same machine was implemented across the remaining three drives. Each drive was divided into two 4.5GB partitions. Drive 0 partition 0 was mirrored on drive 1 partition 1, drive 1 partition 0 was mirrored on drive 2 partition 1, and drive 2 partition 0 was mirrored on drive 0 partition 1. Then guess what? Yep, you guessed it! The three mirrored pair were put together as RAID5 to make a 9GB logical volume. That was the root volume.

--
www.wavefront-av.com
Re:Stick with hardware RAID by julesh · 2004-11-01 23:03 · Score: 1

PHB: Why have you asked for this additional hardware budget?
Sysadmin: Well, we have this disk array that's getting kind of old. We did have a spare controller, but the original died last week and we now can't get a compatible replacement.
PHB: So, why do you need a new controller and eight new disks? Wouldn't just a new controller suffice? Can't you replace the disks on a one-by-one basis as they fail?
Sysadmin: Because if we plug the old disks into the new controller, we won't be able to get the data that's on them. We'll then have to restore from backups, which is a time consuming process that would leave the servers unavailable for an entire day. What we should do is set up a new array now, and copy the data across to it, which is much faster.
PHB: How long is this controller we have likely to last for?
Sysadmin: Well, it's solid state, so it'll probably last at least 5 years.
PHB: I've always said, "if it ain't broke, don't fix it". Come back to me in 3 years when it's starting to look like it might be a problem soon.
Sysadmin: ??

This scenario can be avoided using software raid, however incompetent your PHB is.
Re:Stick with hardware RAID by 1lus10n · 2004-11-04 16:07 · Score: 1

Well ultra2 was new in the mid-late 90's so compare it with software raid running on a 300Mhz box and you will probably see worse performance. Of course it all depends on the setup, having a stand alone box to do RAID is nice, but the bottleneck created by most networks negates the current highend drives. The other thing thats worth noting is what type of connection the card has, the 4200SA's were 64bit PCI so that wasnt an issue. What OS/drivers were you using ? (I obviously do not have a 4200 sitting around to benchmark, but I think that card should have handled more than 7 writes per second.)

--
"Two things are infinite: the universe and human stupidity; and I'm not sure about the the universe." --Albert Einstein
Re:Stick with hardware RAID by amorsen · 2004-11-05 01:45 · Score: 1

Everyone thought it should have handled more than 7 writes per second. This was with Windows NT 4 Server, whatever drivers/firmware was current in 2001. Note that the writes were synchronous, I am sure that performance with asynchronous writes would have been much higher.
The application with the problem was a DHCP/DNS server. It had its own proprietary database, and for each DHCP-address given out it had to do a synchronous write to disk (so it would not reuse addresses). Doing disk profiling showed that the system performed great for 3-4 requests per second (with the 4200, less with the slower controllers), but as soon you went beyond that the writes took a long time to complete. Of course the system should have been using RAID-1 instead of RAID-5, but getting that changed was a long slow process.

--
Finally! A year of moderation! Ready for 2019?

Don't go with 3ware by Codename_V · 2004-10-30 10:47 · Score: 0, Troll

I don't know much about software RAID, but one thing I do know, don't go with 3ware.

--
Free will is just an illusion

Re:Don't go with 3ware by Anonymous Coward · 2004-10-30 10:51 · Score: 0

Why.
Re:Don't go with 3ware by Anonymous Coward · 2004-10-30 10:52 · Score: 0

Just curious, why?

I've had great experience with 3Ware. I trust quite a bit of data to the 8000LP2 cards in pizza box IBM servers.

Thanks.
Re:Don't go with 3ware by Moridineas · 2004-10-30 10:53 · Score: 1

It must be obvious that you donn't know much about software RAID, because 3ware isn't software raid--they make hardware.

And they make quite good cards too, that are highly supported in linux,freebsd, etc. (I have an 8 port SATA raid card in use atm)
Re:Don't go with 3ware by GigsVT · 2004-10-30 10:53 · Score: 2, Interesting

Have you tried the 9500 series? It looks much nicer than their older offerings.

We've run several 7810s, 7850s in the past, totalling quite a few terabytes. All in all it's not too awfully bad, but the cards do seem to have trouble with dropping drives that don't seem to have any real problems (they recertify with the manufacturer's utility often with no errors).

If you go 3ware though, get the hot swap drive cages from 3ware. They are expensive, but it makes it much nicer.

--
I've had enough abrasive sigs. Kittens are cute and fuzzy.
Re:Don't go with 3ware by suwain_2 · 2004-10-30 11:00 · Score: 3, Insightful

This is the worst possible type of advice. Do you have any reason for not using them? Maybe you've bought dozens and they've all blown up and burnt your house down, which would be a good reason to not buy 3Ware. Maybe you work for a competitor.

For all I know, you could have a very good reason. But if you tell someone to make sure to to stay away from something, you should provide a reason. Especially if it's something that seems to have a really good reputation.

--
________________________________________________
suwain_2 :: quality slashdot p
Re:Don't go with 3ware by Dop · 2004-10-30 11:12 · Score: 2, Informative

We've had two different 3ware hardware RAID cards without any problems in the last 3 years.

I've done software RAID as well using Promise IDE controllers. Fortunately for us we never had a drive fail in the software RAID so I can't comment on how difficult it is to recover from a failure.

Interestingly enough, we ran some fairly intense iozone tests on both the hardware and software RAIDs with very little difference in performance (maybe that's why the parent poster doesn't like the 3ware stuff). But... we also ran these same tests with a fibre-channel SAN disk, again with very little performance difference.

Maybe it was a Bus limitation... I didn't have time to investigate it any further.
Re:Don't go with 3ware by fire-eyes · 2004-10-30 11:16 · Score: 1

This is slashdot. Not a legitimate tech forum.

--
-- Note: If you don't agree with me, don't bother replying. I won't read it.
Re:Don't go with 3ware by Codename_V · 2004-10-30 12:52 · Score: 1, Informative

Fine. Didn't realize you slashdot types were so untrusting. =) At my work we initially went with about 12 systems all with assorted 8 port or 4 port 3ware cards. Out of the 12 systems, only 3 of the RAIDs are currently in working order, and I expect they'll fail shortly. Sure they work great at first, but my idea of a nice RAID 5 setup is that when one drive fails you pop a replacement in and you're good to go. With 3ware cards, a drive fails, you reboot your system, and then you can't even boot up until you pull the 3ware card, or pull all the drives off of the card. I've tried and tried to slowly add the drives back but I always seem to end up with an unbootable system. I'll tell you, I'm 100 percent happier with the ide to scsi RAID boxes we now go with.

--
Free will is just an illusion
Re:Don't go with 3ware by pyite69 · 2004-10-30 13:33 · Score: 1

To put in my $.02, we have a dozen servers with > 500 gigabytes each on 7500's and above. They are wonderful controllers, I am going to get one for my home Mythtv box here Real Soon Now.
Re:Don't go with 3ware by Gentlewhisper · 2004-10-30 13:43 · Score: 1

Yeap, you said it, it LOOKS nicer.

But on storagereview.com forums, many are saying how its performance is really sucky.

I have not tried it yet, and will soon be in the market again to move on to the latest PCI-e SATA II controller, but 3ware... I'd pass.

Currently I'm using a LSI Logic card, very good.

LSI Logic are the guys who made Megaraid, so they do know their stuff.

I've also used an Adaptec before, pretty fine card too.

--
Online backup with Mozy, sounds like Ozzie, but more!
Re:Don't go with 3ware by Codename_V · 2004-10-30 14:44 · Score: 1

Oh, and just to clairfy, the thing won't even post with the 3ware card in. What kinda nonsense is that?

--
Free will is just an illusion
Re:Don't go with 3ware by Codename_V · 2004-10-30 14:50 · Score: 1

Cmon jack, you misunderstand me, I'm not saying 3ware is software RAID here, at least I'm sure it's not totally software RAID, although I do suspect a good part of it is done in software just like a winmodem. Anyway, all I'm saying is regardless of what you go with, be it Linux software RAID or some sort of hardware RAID solution, you're best off staying shy of 3ware products (I highly recommend netapp by the way).

--
Free will is just an illusion
Re:Don't go with 3ware by Anonymous Coward · 2004-10-30 16:30 · Score: 0

I have had such a problem in the past, it was fixxed by just moving the card into a diffent pci slot.
Re:Don't go with 3ware by realdpk · 2004-10-30 16:45 · Score: 1

I've had exactly the opposite experience with 3Ware vs. the competition. 3Ware works flawlessly with drive swapping, putting in spares, bringing out spares; no rebooting necessary. This is under FreeBSD, and with the PATA and SATA versions of the card.

I'm not saying you're wrong, but I've run several servers with different 3Ware cards and have had a great experience.
Re:Don't go with 3ware by Fallen_Knight · 2004-10-30 21:51 · Score: 1

your the only person i'd head of haveing any trouble with 3ware, usualy if a product has troubles like that, 9/12 fails, you'll hear about it.

but with 3ware alls i hear is good things witch leads me to belvie its not 3ware but your drives or MB or something else.
Re:Don't go with 3ware by Anonymous Coward · 2004-10-31 04:25 · Score: 0

after working with a certain Tyan motherboard I found out 3ware boards only work well together wird mobos that adhere to the pci standards peferctly.

they have a very high thruput so its understandable that they demand good timing and adherence to specs.
Re:Don't go with 3ware by Moridineas · 2004-10-31 05:07 · Score: 1

Well, FYI then, 3ware is not software raid, not at all, nothing like winmodem. The one I've got in use is a 64-bit pci card that has quite nice performance! It's got excellent cross platform support (their phone techs will actually answer questions about linux and freebsd for instance!)

also it has some quite nice management tools, including a very nice www based monitoring daemon.

Why don't you like 3ware? I've admittedly only put two systems into production with 3ware hardware, but this was after extensive reading and searching, in which 3ware received great reviews.

stick with hardware by Anonymous Coward · 2004-10-30 10:47 · Score: 2, Insightful

Take it from me, stick with a hardware raid 5, reliablity is thru the roof, and cards are now around 300-500 for one with 128 mb of ram. Ince you spent 960 dollars on the harddrives, you might as well trust their organization to something of equal quality.

my 2 cents

Re:stick with hardware by phrasebook · 2004-10-30 12:01 · Score: 1

As usual, no reasons for buying the RAID card are given. Let me guess... you once tried to get software RAID working, but couldn't. Oh well, part of having a RAID is feeling better about your storage, so if the card makes you happy, fine.
Re:stick with hardware by lawpoop · 2004-10-30 15:18 · Score: 1

5 * $120 = $960?

--
Computers are useless. They can only give you answers.
-- Pablo Picasso
Re:stick with hardware by lawpoop · 2004-10-30 15:29 · Score: 1

n/m. My mistake.

--
Computers are useless. They can only give you answers.
-- Pablo Picasso
Re:stick with hardware by Fallen_Knight · 2004-10-30 21:55 · Score: 1

software raid.
windows software raid probaly sucks
linux software always has the chance to be borken in an update or cfg file change/curruption or what not

hardware raid
the software on the card never changes, if it works it will always work (for the most part you hope)
8 drivers now appear as one, simple setup under any OS
and what everyone seems to be missing, is you can dual boot with it and not have a linux only drive.

theres some good reasons i think
Re:stick with hardware by Hast · 2004-10-31 10:45 · Score: 1

the software on the card never changes, if it works it will always work (for the most part you hope)
I'm not sure if you ment to be ironic here; but don't you think it's a bit funny that you buy a piece of hardware because "it never breaks" to protect from dataloss in case another piece of hardware breaks? (Yeah yeah, moveable parts and that.)

And if you've never had hardware fail then you're about due for it. ;-)
Re:stick with hardware by Fallen_Knight · 2004-10-31 21:34 · Score: 1

i've had maxtor drives just about to fail, but i switched to seagate before they did. fried a MB once, but thats baout it, guess i've been lucky

and from what i hear 3ware will send out a replacement and the newer cards reconize the older ones.

Re:How about by Anonymous Coward · 2004-10-30 10:48 · Score: 0

i guess nobody'd notice anything

don't use ext2 by gp310ad · 2004-10-30 10:49 · Score: 1

unless you want to wait forever for fsck

--
Do not look into LASER with remaining eye!

Re:don't use ext2 by CrazyGringo · 2004-10-30 10:57 · Score: 4, Funny

If you're a geek, it's a given that you'll have to wait forever for fsck.

hmmm by johansalk · 2004-10-30 10:50 · Score: 0, Offtopic

I really really really wish to know what he's serving that'd need such amount of gigabytes.

Re:hmmm by Anonymous Coward · 2004-10-30 10:52 · Score: 0

pr0n, of course. And for his friends, no less.
Re:hmmm by ScrewMaster · 2004-10-30 10:55 · Score: 1

VOB.

--
The higher the technology, the sharper that two-edged sword.
Re:hmmm by Anonymous Coward · 2004-10-30 10:57 · Score: 0, Informative

There are alot of things that can take up gigabytes of space. For instance:

mp3s
movies
scat porn
japanese tentacle rape dating simulators
video captures of his nubile, young 14-year old neigbor undressing, oh to view her budding sexuality she is but a flower of femminine innocence
MAME Roms
Re:hmmm by HiyaPower · 2004-10-30 10:57 · Score: 1

Try a dvd collection. 250 dvds at 8 gb per = 2 TB. I've had around 2 TB on a couple of my machines for a long time for that purpose.
Re:hmmm by Lusa · 2004-10-30 10:58 · Score: 1

pr0n cache
Re:hmmm by Anonymous Coward · 2004-10-30 11:05 · Score: 0

pr0n?
Re:hmmm by Deraj+DeZine · 2004-10-30 11:09 · Score: 1

pr0n

--
True story.
Re:hmmm by Anonymous Coward · 2004-10-30 11:11 · Score: 0

pr0n!
Re:hmmm by WoBIX · 2004-10-30 11:56 · Score: 1

Funny I can see.... but Informative??? Some people find inspiration in the oddest things.
Re:hmmm by johansalk · 2004-10-30 12:23 · Score: 1

It strikes me as absolutely bizarre that someone could have such a large collection of porn, unless he's trying to profit from it. What about copyright violations? I doubt that he's produced enough porn to fill up such massive storage.

Here is a better question by PrvtBurrito · 2004-10-30 10:51 · Score: 3, Insightful

Is there a good resource for hardware/software RAID support on linux? Tech support is always a challenge and we have a number of 3ware 8way and 12way powered by 250gb drives. We often have lots of mysterious drops on the array that require reboots or even rebuilding the array. Royal pain in the ass.

--
Laboratree - Scientific collaboration based on OpenSocial.

Re:Here is a better question by GigsVT · 2004-10-30 11:03 · Score: 4, Interesting

I just posted in another thread about 3ware and mysterious drops of seemingly good drives. Even with the ultra-paranoid drive dropping, we have never lost data on 3ware.

Other than that, 3ware has been decent for us. We are about to put into service a new 9500 series 12 port SATA card.

I wish I could say our ACNC SATA to SCSI RAIDs have been as reliable. We have three ACNC units, two of them went weird after we did a firmware upgrade that tech support told us to do, lost the array.

We call tech support and they say "oh we didn't remember to tell you when you upgrade from the version you are on, you will lose your arrays".

--
I've had enough abrasive sigs. Kittens are cute and fuzzy.
Re:Here is a better question by Anonymous Coward · 2004-10-30 12:15 · Score: 1, Informative

Just to follow up on this a little. I used to work with a lot of 1-2TB based 3ware systems. Typically the reason why drives would drop is due to a either a firmware issue on the 3ware or more likely a firmware problem on the Maxtor (I have heard of a similar issue with WD drives). The firmware on the Maxtor would tell the drive to spin down for some reason never to return, but this issue only happened with 3ware based systems. While the 3ware said it was a bad drive it was actually fine and just needed new firmware. While the 3ware is a great solution to get TBs of data on the cheap they are quite slow when it comes to Raid5 especially when rebuilding the array if you do loose a drive. They are great if you are not running a "mission critical" system, but even then the hassle of IDE raid is not worth the savings when you need to manage 10+TB of data.
Re:Here is a better question by Fweeky · 2004-10-30 13:14 · Score: 1

We have similar problems with the aacraid driver and an Adaptec S2120S (4 drive RAID-10). It likes to drop a drive out of the array and then lock up every month or so (and yes, we've replaced card, drive and cable).

This isn't fun when it's your master database to your high volume website. Needless to say, it's now a slave to a FreeBSD master with the same card and drive configuration, which is rock solid. Meh.
Re:Here is a better question by Anonymous Coward · 2004-10-31 06:57 · Score: 0

Mysterious drop outs are usually due to faulty/marginal cabling, or more likely, borderline power supply quality/capacity.

A third possibility is heat (drives running too hot).

Bottleneck by FiReaNGeL · 2004-10-30 10:51 · Score: 1

To me, a cheap 500 Mhz computer (who probably have 64-128 megs of ram, I guess) is gonna choke on 8 250 GB hardrives on Software RAID 5. Some other posters suggested buying an hardware controller, and I agree. If you host stuff for your friends, you can always charge them a little extra to compensate.

--
Eureka Science News - automatically updated

Re:Bottleneck by Anonymous Coward · 2004-10-30 11:31 · Score: 0

cpu really is no issue here, i have a 450 mhz p3 with 6 80 gig ide drives and 512 meg of ram and it serves very well (bout 20-30 MB/s sustained via intel gigabit)

Re:Ok. by satoshi1 · 2004-10-30 10:51 · Score: 2

And are all 15,500 sites about how maxtor drives suck? I guarentee you the aren't. I've been using maxtor drives for as long as I can remember, and not one has failed. I still have a maxtor from five years ago and it's still running fine.

ahem by Anonymous Coward · 2004-10-30 10:52 · Score: 0

file:///usr/doc/raidtools-1.00.3/Software-RAID.HOW TO/Software-RAID.HOWTO.txt

Where I used to work. by suso · 2004-10-30 10:52 · Score: 3, Informative

I used to work at Kiva Networking and we used hardware raid 5 on some machines and software raid 1 and raid 5 on others. Maybe it was just me, but the software raid 5 disks always seemed to last longer. Never much problems with it. In fact, we had more problems getting the hardware raid controller to work with Linux or with buggyness than anything.

Re:Where I used to work. by suso · 2004-10-30 10:54 · Score: 1

By the way, these "machines" that I'm talking about are relatively heavily used mail servers, DNS servers, etc.
Re:Where I used to work. by MicroBerto · 2004-10-31 03:18 · Score: 1

Why don't you work there anymore?

--
Berto
Re:Where I used to work. by suso · 2004-10-31 06:13 · Score: 1

Management reasons, not technical ones.

Works Great! by knitterb · 2004-10-30 10:53 · Score: 1

I have 4 IDE Maxtor 200G drives on two Promise controllers and it's really very stable. I've done this for about the last 4 years for a home network share, with very very good luck.

After setting it all up, I encourage you to attempt to pull a cord from a drive, and boot, and make sure you know how to recover. Nothing can compare to the knowledge of knowing what to do in a serious failure.

My raid5: /dev/md0 576877984 506579124 40995112 93% /data

knitterb@machine%> cat /proc/mdstat
Personalities : [raid5]
read_ahead 1024 sectors
md0 : active raid5 hdk1[3] hdi1[2] hdg1[1] hde1[0]
586075008 blocks level 5, 32k chunk, algorithm 2 [4/4] [UUUU]

unused devices:
0.000u 0.000s 0:00.00 0.0% 0+0k 0+0io 88pf+0w
knitterb@machineb%>

Good luck! :)

--
-bk

Re:Works Great! by Boris226 · 2004-10-30 11:51 · Score: 1

I've been running a Linux software RAID1 on my boot partition and a RAID 5 array for root off of 3 30GB ATA drives. Worked great for ~4 months. While I was away on business (In Australia, no less, I live in the US) I kept getting an error message in my systems logs. I checked it out in /proc/mdstat and one of my drives was down. With one drive down the system ran for the better part of a month while I sat 7000 miles away. When I came back, I powered down the system, replaced the hard drive, booted it back up and synced the array. No problems. The system was a little slow while it ran on 2 drives, but I got no complaints from users. One of the great things about software raid, is that you can set up different RAID types and arrays from different partitions on the same set of drives. I run ext2 for boot and reiserfs for root. Love it and I'll continue to use it until something better and cheaper comes along.

Re:Ok. by zaqattack911 · 2004-10-30 10:55 · Score: 0, Flamebait

UUuuh try googling that phrase with quotes around it genius...

maxtor sucks actually gives 233 search results.. in the world :)

That pretty much means fuckall for your argument.

Software raid by Anonymous Coward · 2004-10-30 10:56 · Score: 0

Just be carefull about what you use as your boot disk. You don't want the diskc containing /boot to be mixed in with the raid.

Re:Stick with hardware RAID, mod this up! by Anonymous Coward · 2004-10-30 10:56 · Score: 2, Interesting

So true. In many cases HW RAID doesn't offer any advantage over software RAIDs and it's only a one new part that can broke and cost $$$ and time to replace.

Moderators, mod this up!

Re:Did you read the RAID-Howto by tokki · 2004-10-30 10:56 · Score: 2, Insightful

The guy could be looking for people's experiences rather in additional to any technical documentation, which is not only smart, but the hallmark of a responsible sysadmin (where knee-jerk comments tend typically aren't).

Re:How about by Anonymous Coward · 2004-10-30 10:56 · Score: 0

PRESENT IT!

I'm doing this as well, my experiences: by pythas · 2004-10-30 10:57 · Score: 2, Informative

I have a smaller array, but it's been largely trouble free.

However, when a drive did drop off line, unless things are on their own channel, it seems to knock off the entire IDE channel as well. It ended up taking the better part of a day to get everything back online again, without any data loss.

It even seems like any time there's an IDE hiccup, you can knock your array offline.

It's definitely cheaper than hardware RAID, and I haven't noticed any performance problems, but sometimes the stability of good old SCSI raid is something I miss. :(

Re:Ok. by Anonymous Coward · 2004-10-30 10:57 · Score: 0

I've had one Maxtor drive fail on me, and one of my friends has had at least 2 fail. I'm pretty happy though - Maxtor replaced my 80gb drive for a 250gb :)

Re:Ok. by pairo · 2004-10-30 10:57 · Score: 1

Results 1 - 10 of about 85,800 for slashdot sucks.
You get what you pay for. :-)

Devil in the... by buddha42 · 2004-10-30 10:57 · Score: 1

Does anyone know if software raid-5 in linux uses a chunk of memory specifically for the write cache? Most hardware raid5 cards have memory for this so that the card can tell the OS everything is done being written and then flush it out of cache at its convinience. If software raid-5 actually relies on the writes to be finished, it would be drastically slower (for writes).

Re:Devil in the... by mortonda · 2004-10-30 11:17 · Score: 2, Informative

the card can tell the OS everything is done being written and then flush it out of cache at its convinience.

Which is absolutely horrible. This violates protocol - mail MTA's demand that data is written to disk before they acknowlege delivery. They get this from the confirmation from the kernel, but if the disk array lies about it, a power failure could lose data even though the kernel assumed it had bee synced properly.
Re:Devil in the... by IgnorantSavage · 2004-10-30 14:14 · Score: 1

Note that higher-end hardware RAID controllers use non-volatile RAM (usually battery powered) for the write-back cache. This prevents the protocol violation since it can survive a power problem so long as power is restored before the NV cache battery goes out. There are also redundant solutions with two RAID controllers that mirror their battery-backed cache to provide even higher reliability. The theoretical MTBF for a properly configured system with one of these is high enough that software bugs are much more likely to cause data loss than any combination of hardware failure.
Re:Devil in the... by mortonda · 2004-10-30 16:24 · Score: 1

Yeah some more expensive controllers can do that, but I don't think you'd hook that up to a system powered by a 500Mhz CPU. :)

One other problem with write cache (that maybe the powered RAM cards fix) is when rebooting - I have seen a system go down for reboot, and hits the POST initialization before the controller is finished syncing the disks... and then it gets reset. Poof! lost data. :(
Re:Devil in the... by Anonymous Coward · 2004-10-30 21:08 · Score: 0

Which is just another reason that SMTP is not a guaranteed delivery mechanism! I wish people would realise this and not use email for something that they consider "ultra critical".
Re:Devil in the... by mortonda · 2004-10-31 09:57 · Score: 1

Quite true, if it is super critical, a followup call is a must. Nevertheless, a hard drive should not lose data that was reported by the kernel to be synced to disk.

Depends on the HDD interface by mattbee · 2004-10-30 10:57 · Score: 1

As an ISP building servers for our own use, we've never had problems with Linux RAID-5 but plenty of problems with recalcitrant disc interfaces that won't easily support hot-swapping and/or clean reporting of drive failures. Even 3Ware cards (until relatively recent firmware revisions) could cause system crashes before they cleanly reported a drive as failed with Linux RAID-5 sitting on top (yes, Linux software RAID-5 is much faster than the hardware RAID-5 with their 8000 series cards). So my advice is just to test a failure: try unplugging a drive from a test array while it's running and see what happens. The "plug & play" aspect might not work as advertised but you shouldn't lose any data through it (we've not).

--
Matthew @ Bytemark Hosting

Works Great! by ogre7299 · 2004-10-30 10:57 · Score: 2, Informative

I've been using RAID 5 with a 3 18 GB SCSI drive setup for about 6 months now, it works very fast and reliably.

The best advice I can give is to make sure each drive has its own channel if you are on standard ATA, you didn't specify SATA or regular ATA. If you're using SATA then all the drives get their own channel by design. If you have more than one IDE device on a channel in a RAID, performance will suffer because IDE can't write to both devices on the channel simultaneously.

software raid it fine by Anonymous Coward · 2004-10-30 10:58 · Score: 0

its money the cheap raid controllers are worthless if you want blinding performance buy the good stuff read 400 or 500 bucks as for the software raid its fine it works its fairly painless but it is no substitute for a good backup. the performance difference between little 'promises' and software is nil.

Vinum with FreeBSD by Anonymous Coward · 2004-10-30 10:58 · Score: 3, Informative

While it's not Linux, I've been using Vinum with FreeBSD for about 3 years with RAID 5 and have never had any problems. My current box is an old VIA 600MHz C3 with FreeBSD 4.8 and a measly 128MB of RAM. As far as benchmarks go, my RAID seems to blow away all of the cheapy hardware cards performacewise as well.

BTW, I switched from Linux to FreeBSD for the server years ago for the stability.

Go hardware... by loony · 2004-10-30 10:59 · Score: 1

If you got the money, go hardware... I have two systems that are identical with the exception that in one I broke one of the ide connectors of the 3ware controller and had to go software raid instead. I have 6 120GB disks in each, on separate channels... There is no problem with the stability and both setups survived at least 2 disk failures each.

Unfortunately the performance is another issue - I get three times the throughput on the 3ware controller. So the $300 I spent on the controller (used) was well worth it...

Peter.

Setting up software RAID is really easy.. by Anonymous Coward · 2004-10-30 11:01 · Score: 0

Enable RAID support in kernel. And install raidtools. Then raithotadd, raidhotremove, raidstart, raidremove, cat /dev/mdstat, /etc/raidtab , 0xFD (Linux raid auto) fdisk partition type are the keywords (on Debian).

Don't run software raid... by TyrranzzX · 2004-10-30 11:01 · Score: 0

It's a poor solution for raid, since if the OS goes, there goes your raid. If you use hardware, at least it'll autodetect.

Mabye I'm wrong and the people have a good solution for it, but still. Personally, if it's a 500mhz box, I'm guessing you're going to be using ATA133 PCI cards along with it, and frankly, raiding those won't net you much more speed. SCSI is what gets you the speed, but as it is you're looking at probably a max of 50 or 60MBps output.

I'd just stick a 100mbps card on it, hook up all 8 drives via 2 IDE ATA133 PCI (2 controllers a card) cards, with the OS installed on a raided partition spread across 4 of the disks, and backed up onto 4 other drives. Setup a cheap batch script that backs up all the files onto a second set of disks when the machine isn't being used, or use a backup program, as well as scanning for data errors/loss every week or so. It works if all it's doing is serving files. Additionally, if you set the "primary" drive as master, and the "backup" drive as slave, and you only use the primary drives for serving data, you'll have no problem.

Really, I'd think Raid Lv5 for 8 drives would be a bit excessive and/or inefficient, considering the PCI bus only gives so much bandwidth. Now, if we were talkin SCSI raid with 64 bit, 66mhz PCI, then your solution would be a good one.

--

Candy-Coated Knowledge

Re:Don't run software raid... by mortonda · 2004-10-30 11:28 · Score: 3, Informative

It's a poor solution for raid, since if the OS goes, there goes your raid. If you use hardware, at least it'll autodetect.

Um, that's bogus. If your OS goes (probably due to hardware?) then you can simply put the drive in a new computer (same basic master/slave setup) and away it goes. Linux knows how to detect its own RAID arrays!

OTOH, if you have a hardware RAID, good luck getting tech support, especially if they no longer carry that board, or have gone out of business altogether.

At least with Software RAID, your data is not stuck in a proprietary format.
Re:Don't run software raid... by megabeck42 · 2004-10-30 14:56 · Score: 2, Interesting

The woeful inaccuracy of your post is really, really painful. Allow me to rebuke.

First of all, linux software raid has excellent autodetection. You need to set the partition identifier to 0xFD so that the autodetector can identify it. As many have mentioned, software raid has a huge advantage over hardware raid for recovery - you can disconnect the drives from one computer, hook them to another and the autodetect code will figure it out. I know this works because I've done it.

Second, for 8 drives and 2 controllers a card, you'd want four ATA133 adapters. Each adapter has, as you said, 2 controllers. You don't want to sue the slave channel, because that will definately kill performance.

Third, Don't install the OS on the raided partition. Don't keep anything fragile or irreplacable on the OS partition. If you want to backup the configuration, backup the configuration. There's no need to raid your boot drive, and if your boot drive fails you can trivially reinstall.

A cheap batch script is not an effective backup solution. What if files are locked or a file is backed up midway through a transaction? I readily agree that RAID is not a backup solution, but to putting any faith in a "cheap batch script" is profoundly naive.

RAID5 has the advantage that you only lose one drives worth of space to parity information. With eight 250 gig drives on a P3 500, its readily obvious that his goal is to inexpensively store a large amount of data with an effective mitigation against a single drive failure. Software RAID5 is an excellent solution for him.

Lastly, I'd recommend one of the intel gigabit cards, because although the drives will only read 50 or 60 megabytes/s, the whole point is moot if your network connection maxes out at around 10 megabytes a second. The client adapters, like the 1000MT is more than enough, and not that expensive.

--
fnord.
Re:Don't run software raid... by Fallen_Knight · 2004-10-30 21:58 · Score: 1

with software raid its stuck in a OS spec format...

from what i hear 3ware has good customer support. SO it seems do your research and find a company you can depend on

Don't screw around - hardware is better. by ErikTheRed · 2004-10-30 11:01 · Score: 5, Informative

Software raid is fine for simple configurations, but if you want to "do it right" - especially considering that you just dropped about a kilobuck on HDDs, go Hardware. A good, reasonably priced true hardware RAID controller that will fit the bill for you is the 3Ware Escalade 7506-8. It has 8 IDE ports, 1 for each drive - you don't want to run two RAID drives in master/slave mode off of a single IDE port; it will play hell with your I/O performance. It's true hardware raid, so you don't have to worry about big CPU overhead and being able to boot with a failed drive (a major disadvantage to software RAID if your boot partition is on a RAID volume, certain RAID-1 configurations excepted). You can buy them for under $450. provantage.com price is $423.48 (I have no relationship with them other than I've noticed that their prices tend to be decent).

--

Help save the critically endangered Blue Iguana

Re:Don't screw around - hardware is better. by slashdot.org · 2004-10-30 11:43 · Score: 1

$315 on eBay right now: right here

Not affiliated either, just a 10 second froogle search.
Re:Don't screw around - hardware is better. by Hrunting · 2004-10-30 11:50 · Score: 3, Interesting

Software raid is fine for simple configurations, but if you want to "do it right" - especially considering that you just dropped about a kilobuck on HDDs, go Hardware. A good, reasonably priced true hardware RAID controller that will fit the bill for you is the 3Ware Escalade 7506-8. It has 8 IDE ports, 1 for each drive - you don't want to run two RAID drives in master/slave mode off of a single IDE port; it will play hell with your I/O performance. It's true hardware raid, so you don't have to worry about big CPU overhead and being able to boot with a failed drive (a major disadvantage to software RAID if your boot partition is on a RAID volume, certain RAID-1 configurations excepted). You can buy them for under $450. provantage.com price [provantage.com] is $423.48 (I have no relationship with them other than I've noticed that their prices tend to be decent).

Hardware RAID5 is fine if your sole goal is reliability. If you need even an iota of performance, then go with software RAID5. The 3wares have especially abysmal RAID5 performance, specially older series like the 75xx and 85xx cards. 3ware's admitted it, and something targeted for fixing in the 95xx series (haven't gotten my hands on those yet, so I don't know).

As for software RAID reliability, I find that Linux's software RAID is much more forgiving than even the most resilient of hardware RAIDs. I've lost 4 drives out of a 12 drive system at the same time, and Linux has let me piece the RAID back together and I've lost nothing. Was the machine down? Yes. Did I lose data? No. Compare that with a 3ware hardware RAID system where I lost 2 drives. Even thought I probably could have salvaged 99% of the data off that array, the 3ware just would not let me work with that failed array.

Also, on any reasonably modern system, the software RAID will be faster. You just have a much faster processor to do the RAID processing for you. The added overhead of the RAID5 processing is nothing compared to a 1-2GHz processor.
Re:Don't screw around - hardware is better. by nfsilkey · 2004-10-30 12:01 · Score: 1

Funny. I had dozens of persons recommend 3Ware as the undisputed kings of hardware RAID controllers. I bit the bullet and bought an 8506-12 SATA for home use (2TB usable RAID-5 w/ a hot spare ... dont ask ;).

Worked great for a few months. But the controller then began to sporadically report drives "Not In Use" during initialization on boot. To make matters worse, the card would sometimes hang during initialization, timeout ("bios not installed"), and fail to be accessable by the kernel once booted. These problems persist to this day. Many calls to 3Ware have resulted in the same verdict: "not enough or dirty power". I dont have the tools to pinpoint dirty power problems, but I have tried to gut the box as best as I could to solve any power consumption problems. This was rather tough given the box's power needs (Athlon XP 2500+, 4 x 128 2100 DDR, 1 x WD 80GB [WD800JB], Sony EIDE DVDROM, 3Ware 8506-12 SATA, 12 x Maxtor 250GB 7200RPM EIDE [6Y200P0], 12 x HPT EIDE -> SATA bridges, ~45 40mm-80mm fans). So I came to accept the shady verdict from 3Ware about power; the box was a beast even for an Antec 550.

Whats more perplexing is that I have built a similar box at work, but with much more conservative power needs (Athlon XP 2500+, 4 x 128 2100 DDR, 3Ware 8506-8 SATA, 4 x WD2000JD). With only 4 drives hanging off an Antec 550, I BEGAN TO SEE THE SAME PROBLEMS. Strange indeed. The answer from 3Ware was the same: "not enough or dirty power". Well, "not enough" sounds like horseshit to me.

Its important to remember that I evaulated software RAID as a solution; more people recommended it over hardware/3Ware. The problem I had was interfacing the drives to one board! Certain add-on PCI controllers, such as those offered by Promise, can only have 1-2 per system. Software RAID-5 without drives on their own channels is certain death. So the challenge on how to get your drives all hanging off their own channel within one box is tough when you want 6-12 drive arrays.

If I could do it all over again, I would forget about my small penix and build me a couple software RAID-5 boxen with ~1TB per box. No power problems, no 3Ware hassles, no mega-drives/box issues. Oh well... live and learn.
Re:Don't screw around - hardware is better. by mprinkey · 2004-10-30 12:08 · Score: 1

Honestly, it is a common practice to buy 3ware "raid" controllers and only use them as non-raid IDE controllers. Then use Linux Software RAID to build the array.
Re:Don't screw around - hardware is better. by Anonymous Coward · 2004-10-30 12:10 · Score: 0

Athlon XP 2500+, 4 x 128 2100 DDR, 1 x WD 80GB [WD800JB], Sony EIDE DVDROM, 3Ware 8506-12 SATA, 12 x Maxtor 250GB 7200RPM EIDE [6Y200P0], 12 x HPT EIDE -> SATA bridges, ~45 40mm-80mm fans

Christ almighty!

Oh well... live and learn.

And spend.
Re:Don't screw around - hardware is better. by RyuuzakiTetsuya · 2004-10-30 12:10 · Score: 1

3Ware Escalade 7506-8

Can I get it with gold plated spinner rims?

--
Non impediti ratione cogitationus.
Re:Don't screw around - hardware is better. by ErikTheRed · 2004-10-30 12:38 · Score: 1

Three words: External Drive Enclosures.

Spread that power load arouund, dude. I'm surprised your power supply hasn't yet melted into a pile of slag.

--

Help save the critically endangered Blue Iguana
Re:Don't screw around - hardware is better. by MourningBlade · 2004-10-30 13:01 · Score: 1

considering that you just dropped about a kilobuck on HDDs
Man, they're always doing funny prices like $24.95, $13.78, etc. Now they're doing powers of two?
Damn. Well, $5.12 doesn't buy what it used to. Just my $0.02
Re:Don't screw around - hardware is better. by Jeff+DeMaagd · 2004-10-30 14:12 · Score: 1

How well does software RAID work on a dual processor system? In the past, it was basically asking for problems, so a dual CPU system pretty much needed the hardware for best reliability.
Re:Don't screw around - hardware is better. by sjames · 2004-10-30 14:49 · Score: 1

RAID 5 made a lot of sense when HDs ran $50-$100/ GB. Now that storage is dirt cheap, it makes more sense to go with RAID 1 or 0+1 and do soft RAID ($423 will buy a lot of storage these days). The big CPU overhead comes from RAID 5.

Hard RAID makes sense primarily when the I/O bandwidth with soft RAID would exceed your PCI bus. Fileservers with less than 3 maxed out GigE connections need not apply, the network will be the bottleneck there.

Strongly agreed on the master/slave arrangement though.
Re:Don't screw around - hardware is better. by k12linux · 2004-10-30 16:54 · Score: 1

Hard RAID makes sense primarily when the I/O bandwidth with soft RAID would exceed your PCI bus.

That seems counter-intuitive to me. If drives off multiple cards exceed your PCI bus, how would it not be true for a single card? The only explanation I could think of is if you were running so close to bus speed that the extra parity data put you over. If your box has multiple PCI buses, you could even do something that he RAID card wouldn't allow... like split the traffic accross multiple busses.
It seems like it would be more of an issue with the IDE bus since virtually all IDE hardware RAID cards give each drive it's own bus. The solution, of course, is to not put multiple drives on one IDE bus. (Doing so is stupid anyhow.)
Re:Don't screw around - hardware is better. by adolf · 2004-10-30 17:47 · Score: 1

$450, and the only[1] advantage you've stated is that it doesn't have "big CPU overhead." At best, this statement is misleading.

It has been the case for years now that software RAID requires very little CPU. Meanwhile, $450 buys a fuckton of Intel/AMD CPU, and that fuckton will always be bigger than the shovelfull of CPU provided on a RAID card.

After that, it's just math - and simple math, at that. (Calculating parity is not difficult, mmkay?)

Thus, spending the money on a faster processor will, AFAICT, always yield higher total system performance than using hardware RAID.

Why would it be more advantageous to spend $450 on a proprietary RAID adapter, than to invest the same amount of money in CPU power and get a faster system?

[1] I'm not aware of any present limitations in Linux's handling of software RAID that would preclude the use of a redundant boot partition. Correct me if I'm wrong, but I've been using one here for some time now.

--
Kid-proof tablet..
Re:Don't screw around - hardware is better. by photon317 · 2004-10-30 17:48 · Score: 3, Informative

Don't forget that hardware raid is a single point of failure. The best solution for the absolute best redundancy and performance is software raid set up to be fault tolerant of controller failures. For example, put two seperate scsi cards in the box, and software mirror your data between them, and then stripe on top of that for added performance if you have the drives. When using striping and mirroring together, always mirror at the lowest level, then stripe on top of that.

The basic idea is:

C == controller
D == disk
R == virtual raid disk

C1 --> D1,D2,D3
C2 --> D4,D5,D6

R1 = mirror(D1,D4)
R2 = mirror(D2,D5)
R3 = mirror(D3,D6)

R4 = stripe(R1,R2,R3)

--
11*43+456^2
Re:Don't screw around - hardware is better. by elandal · 2004-10-30 18:49 · Score: 1

3ware 9508S with 5 Maxtor DM+9 200GB disks in RAID5 configuration under WinXP has typical read-speed of about 40MB/s when you have several apps accessing the disk at the same time, and clocks a little over 100MB/s when you close the apps and benchmark the system.
Don't know if that's fast or not - just that it's enough for me.
Re:Don't screw around - hardware is better. by Fallen_Knight · 2004-10-30 22:04 · Score: 2, Informative

http://www.tomshardware.com/storage/20040831/sata- raid-controller-12.html
http://www.tomshardware.c om/storage/20040831/sata- raid-controller-16.html

seems plenty fast to me, guess they fixed the problems, at elast enough to be at the top:)
Re:Don't screw around - hardware is better. by sjames · 2004-10-31 03:32 · Score: 1

That seems counter-intuitive to me. If drives off multiple cards exceed your PCI bus, how would it not be true for a single card?

Because in soft RAID, the driver must queue the write to each drive seperatly and the block is DMAed over the PCI once for every involved drive. In hard RAID, the card recieves the block once over the PCI and then issues the write to each drive from it's own buffer.

If your box has multiple PCI buses, you could even do something that he RAID card wouldn't allow... like split the traffic accross multiple busses.

I didn't explicitly go into that, but in that case, you are presumably NOT hitting the I/O bandwidth limit. If you are, a hybrid approach where you use 2 hardware raids striped together in a soft RAID will help alleviate the problem.
Re:Don't screw around - hardware is better. by TheOrquithVagrant · 2004-10-31 04:58 · Score: 1

I've been running linux software raid5 on numerous SMP systems for over 4 years, and I've never seen any trouble with it, either under normal operations, or after HD or controller failures. Might be filesystem-specific, though - I've used ext2, ext3 & xfs on these machines, never jfs or reiserfs, so I can't vouch for their behavior on this setup. Likewise, all the SMP systems have been x86-architecture. No experience with PowerPC, Sparc or Alpha SMP systems running linux software raid.
Re:Don't screw around - hardware is better. by Skuld-Chan · 2004-10-31 08:38 · Score: 1

If you knew so much about 3ware you'd know that you can force a rebuild on a degraded raid 5 - even if you've lost more than 2 drives. Even my cheapo 7540-4 can do this.

3ware also supports hot spares - so why you'd end up having to do this on an enterprise system is beyond me.

Avoid cheap raid controllers by Alan+Cox · 2004-10-30 11:02 · Score: 3, Insightful

The cheap raid controllers are almost always software raid and not worth it. If performance is critical some of the higher end SATA and SCSI raid stuff is worth it, but a lot of that sucks too so do benches, take recommendations and don't believe in brand names...

Re:Avoid cheap raid controllers by ericdano · 2004-10-30 11:29 · Score: 1

What? No. You can get very decent Hardware RAID5 controllers, and the performance is a whole lot better than software.......

--
It's either on the beat or off the beat, it's that easy.
I moderate therefore I rule!
--
Re:Avoid cheap raid controllers by realdpk · 2004-10-30 11:59 · Score: 1

The only good RAID cards I've run in to are the 3Ware IDE cards (ATA and SATA). Adaptec's SCSI RAID stuff is awful in comparison. Commnication with the card is flakey at best, it handles failures *very* poorly (drive disappears, some data may still be lost, because the RAID may crash. drive replaced (zapped) may cause SCSI errors as well), it doesn't seem to do staggered-spinup (so be prepared to trip breakers, although they may have added that feature later).

The 3Ware cards are about the same price, maybe a bit more for the SATA, but it's well worth it. PATA doesn't compare performance-wise, but if you're just doing storage, they're a good choice too.

Whatever you do though, avoid Adaptec.
Re:Avoid cheap raid controllers by Jennifer+E.+Elaan · 2004-10-31 03:02 · Score: 1

Oddly enough, we took advantage of that in our household. We bought a Promise TX4 4-port SATA "RAID" card, knowing that it only pretended to be a hardware RAID through tricks of the BIOS and proprietary driver. It was a simple matter to reflash the card's BIOS with the 0xAA55 boot signature removed (a bit of hexediting later), disabling the option ROM and turning it into a standard 4-channel SATA controller. We've been running a Linux software RAID on it since then across 4 160GB Maxtor drives, and overall we are very impressed with it. Performance is good, and it worked flawlessly through a drive failure and replacement.

--
Hardware, software, and blinking lights!

*DO* go with 3ware by Alan+Cox · 2004-10-30 11:04 · Score: 2, Informative

Except for the early 7000 series they are good cards and have decent performance too. I'm very very happy with the 3ware I have even though its one of the quite early designs.

Re:*DO* go with 3ware by GigsVT · 2004-10-30 11:20 · Score: 1

Does anyone have any caveats with the new linux kernel driver for the 3w-9500 series?

We are about to go live with a new series card, and so far it looks a lot nicer, it has a buzzer when drives go bad, instead of relying on 3DM to warn you, and a 12 disk 2.8TB RAID 5 array built in about 5 hours.

I guess we are also going to get to test the LBD stuff for >2TB filesystems. Fun fun. :)

--
I've had enough abrasive sigs. Kittens are cute and fuzzy.
Re:*DO* go with 3ware by phoxix · 2004-10-30 13:30 · Score: 1

What is wrong with the 7000 series ?

thanks

Sunny Dubey
Re:*DO* go with 3ware by tf23 · 2004-10-30 14:04 · Score: 1

dunno. my 7800 works just fine... for the past few years infact :)

--
http://slashdot.org/~tf23/journal
Re:*DO* go with 3ware by Codename_V · 2004-10-30 14:39 · Score: 1

Sweet. Alan Cox has my back. Down with 3ware.

--
Free will is just an illusion
Re:*DO* go with 3ware by BigGerman · 2004-10-30 14:46 · Score: 1

I have a 7000 with two drives in raid-1. Just the other week one of the drives failed (after two years). The array did not behave correctly; it should have remained operational on one drive but instead it started producing flackey results (like some files partially available) and it was hard to diagnoze the problem. It did rebuilt correctly after I replaced the drive.

Bottleneck is not CPU by mortonda · 2004-10-30 11:05 · Score: 2, Insightful

If all it does is serve files, it should do fine. The 500Mhz is not going to be a factor at all, in fact, the CPU will be idle most of the time. The real thing to optimize in a file server is the ATA bus speed and hard drive latency.

Or, have two backup RAID controllers. by Futurepower(R) · 2004-10-30 11:06 · Score: 4, Informative

This is a VERY big issue. We've found that Promise Technology RAID controllers have problems, and the company doesn't give tech. support when the problems are difficult, in our experience.

--
Government data compares Democrat and Republican economics.

One Drive per controller by mcleodnine · 2004-10-30 11:07 · Score: 3, Insightful

Don't hang a pair of drives off each controller. Get a truckload of PCI ATA cards or a card with multiple controllers. Don't slave a drive. (No, I do NOT know what the correct PC term is for this).

Also, give 'mdadm' a whirl - a little nicer to use than the legacy raidtools-1.x (Neil's stuff really rocks!)

Software RAID5 has been working extrememly well for us, but it is NOT a replacement for a real backup strategy.

--
one better than mcleodeight

Hot spares by lathama · 2004-10-30 11:08 · Score: 1, Informative

Declare at least one hot spare. I would declare two for your setup but YMMV.

nr-spare-disks 1

device /dev/hdh1
spare-disk 0

--
The GPL, for those that truely understand.

Performance Tips by Alan+Cox · 2004-10-30 11:09 · Score: 4, Informative

There are a few things that really help in some cases, but RAM isn't always one of them.

If you've got a lot of data that is read/re-read or written/re-read by clients then RAM really helps, streaming stuff which doesn't get many repeat accesses (eg running a movie editing suite) it might not help at all

For performance its often worth sacrificing a bit of space and going RAID 1. Again depends if you need the space first or performance first.

Obviously don't put two drives of a raid set on the same IDE controller as master/slave or it'll suck. Also if you can find a mainboard with multiple PCI busses that helps.

Finally be aware that if you put more than a couple of add on IDE controllers on the same PCI bus it'll suck - thats one of the big problems with software raid 5 versus hardware which is less of a problem with raid 1 - you are doing a lot of repeated PCI bus copies and that hurts the speed of drives today.

I use raid1 everywhere, disks may be cheap but you have to treat them as unreliable nowdays.

Be careful with "hotplugging" by mortonda · 2004-10-30 11:09 · Score: 1

IDE interfaces, and some SATA interfaces, are not designed to be hotplugged. There are special electrical circuits on hot pluggable scsi drives.

Once you yank the plug on an IDE, you *must* power down the system before plugging it back in. Sometimes yanking the plug hangs the kernel.

Power down the system, pull a drive, and then start it up. It should detect the missing drive on bootup. Alternately, follow the instruction on the LINUX RAID HOWTO. You have read that, haven't you?

Re:Be careful with "hotplugging" by cymen · 2004-10-30 12:32 · Score: 1

That's why you only yank out the power plug when the system is on. Then you power down and plug it back in (unless you like to live dangerously). Obviously, this is not something you do to a production system.
Re:Be careful with "hotplugging" by voidptr · 2004-10-31 07:22 · Score: 1

4 pin molex connectors on most drives aren't designed to be hot plugged either, and if you're not lucky, they'll arc when you unplug them and can do quite a bit of damage.

--
This .sig for unofficial government use only. Official use subject to $500 fine.

RAID5 by mikewelter · 2004-10-30 11:09 · Score: 2, Interesting

What will you connect eight drives to? Four PCI ATA controllers? I have eight 200GB drives on my data server using a 3Ware RAID controller, and it has worked wonderfully for 18+ months. I have had a drive fail (due to insufficient cooling), and the system didn't even hiccup. I have a software RAID system at a client's location. Whenever there is a power failure, the system comes back up nicely. However, because of the abnormal shutdown, the software RAID tries to recover one of the disks. This absolutely eats the processor for 16 hours. 98-100% utilization. Fiddling the /proc parameters is no help. I think this is a bug--what could it be doing for 16 hours?

RAID-5 with 6x 300GB Maxtors by EvilMonkeySlayer · 2004-10-30 11:10 · Score: 1

I built a fileserver for work on the cheap, it's been up for about a month now without any problems.

You have two choices with software RAID under linux, raidtools and mdadm.

Raidtools is a bunch of tools for setting up, looking after etc a software RAID under linux and mdadm is essentially the replacement to Raidtools, everything is built into the single mdadm program.
I personally find Raidtools to be easier to use at the command line, however mdadm is the future. Also of note, if you want to set it up easilly without the hassle just use Webmin, its RAID setup is a breeze.

RAID isn't totally reliable by Anubis333 · 2004-10-30 11:11 · Score: 1

If you are concerned with reliability, don't store everything in one device that can get taken out by fire/electricity/anything. Parity also isn't very reliable, I had two drives go bad in my RAID 5 and I was screwed. Also, Maxtor drives arent reliable, not like SCSI drives.

And as far as IDE drives, Seagate makes 200GB drives with 8mb cache, 5 year warranties, and they were at Frys last week for 50 bucks.. (rebate)

Re:RAID isn't totally reliable by Anonymous Coward · 2004-10-30 12:14 · Score: 0

Maxtor drives arent reliable, not like SCSI drives

What about Maxtor's SCSI drives? :-)
Re:RAID isn't totally reliable by tf23 · 2004-10-30 13:39 · Score: 2, Informative

I had two drives go bad in my RAID 5 and I was screwed

I take it you didn't have a drive sitting waiting as a hot-spare?

I got bit by this once. Never again.... now I always have a hotspare waiting to jump into place for an instant rebuild.

--
http://slashdot.org/~tf23/journal

My Configuration - RAID5, Mandrake 10, Athlon 2000 by Luciq · 2004-10-30 11:11 · Score: 1

I have a very similar setup, but I'm currently only using three hard drives. I'm running Mandrake 10 on an Athlon XP 2000 with three Western Digital 250GB drives. That gives me 500GB of usable space, which I'll probably upgrade to ~1TB (5 drives) in January. So far I've had no problems. I have a UPS, but I've pulled the plug several times for testing purposes with no ill effects (I tested the setup for several weeks before putting real data on it). I think a big reason this setup works for me is that the server is never under much stress (serving 3-4 household PCs), and it's only used as a file server. I use computers for everything. Work projects, personal projects, audio/video editing, etc. I never delete anything and would be devastated by a data significant data loss. So I use RAID5 with monthly backups (unfortunately backups are still important), and all important data is saved on the storage server. I'd recommend against sticking 8 drives in a box. If you're set on using 8, you might want to consider putting a few in external enclosures. Make sure you keep cool air moving across them to avoid shortening the lifespan of your drives. Currently, I'm only using about 250GB on my 500GB server, with most of the space being consumed by data from my audio/video projects. Unless you're actually going to use 1750GB of space, I suggest using five drives or so in the server while using the remaining three for backups. Also as was mentioned in a previous post, your system may not be beefy enough to handle an 8 drive RAID5 array. Let us know what you end up doing.

My Vote: Use Hardware for RAID 5 setups by Whizzmo2 · 2004-10-30 11:11 · Score: 1

As other posters have mentioned, software raid is fine for RAID 0, 1, 0+1. As you get to RAID 3,RAID 5, and RAID 6, however, your processing requirements go up quite a bit.

A SATA RAID 5 card with hardware XOR engine and a DIMM slot for cache might be a cost-effective option for you. (Goes for ~$180 on Pricewatch, or ~$240 on Dealtime)

Oh, and I would have goine with HGST, Western Digital, or Seagate for your drives... but I suppose hardware failure is what RAID 5 is for :)

CONFIGURE IT RIGHT!! small parts... by brak · 2004-10-30 11:13 · Score: 5, Informative

You will get responses from people with good and bad experiences, but they are all jaded by their small particular case. After seeing what can happen with dozens of machines (8 drive and 4 drive) running Linux software RAID5, here is some concrete advice.

First, ensure that all of the drives are IDE masters. Don't double up slaves and masters.

Secondly, DON'T create gigantic partitions on each oft he 250's and then RAID them together, you will get bitten, and bitten hard.

Here's the skinny...

1) Ensure that your motherboard/IDE controllers will return SMART status information. Make sure you install the smartmon tools, configure them to run weekly self tests, and ensure you have smartd running so that you get alerted to potentially failing drives ahead of time.

2) Partition your 250GB drives into 40 GB partitions. Then use RAID5 to pull together the partitions across the drives. If you want a giant volume, create a Linear RAID group of all of the RAID5 groups you created and create the filesystem on top of that.

Here's why, this is the juice.

To keep it simple, let's say there are 20 secotrs per drive. When a drive gets an uncorrectable error on a sector, it will be kicked out of the array. By partitioning the drive into 5 or 6 partitions, let's say hd(a,c,e,g,i,k,l)1 are in one of the RAID5 groups, which contain sectors 1-4 (out of the fake 20 we made up earlier)

If sector 2 goes bad on /dev/hda1, Linux software RAID5 will kick /dev/hda1 out of the array. Now, it's likely that sector 11 might be bad on /dev/hdc. If you hadn't divided up the partitions, you would lose a second disk out of the array during a rebuild.

By partitioning the disks you localize the failures a little, thus creating a more likely recovery scenario.

You wind up with a few RAID5 sets that are more resilient to multiple drive failures.

If you are using a hot spare, your rebuild time will also be less, at least for the RAID5 set that failed.

I hope this makes sense.

My advice to you is to bite the bullet and simply mirror the disks. That way, no matter how badly they fail you'll have some chance of getting some of the data off.

OS X software raid by v1 · 2004-10-30 11:14 · Score: 1

After some initial hicups with the system in Mac OS 10.2, the new 10.3 seems to have a usable software RAID solution. It supports just about any method of connecting the drive to the computer (short of network) and supports most any kind of storage device. It doesn't support RAID5 yet though, just stripe and mirror. I've got four mirrored volumes up on my server, and have had few problems with them. Rebuilding must be done with the volume offline. It's not a perfect solution yet, but it's a nice free alternative for your OS X systems.

I've also tried CharismacRAID, and omg... STAY AWAY from this. After the array crashed for the third time for no apparent reason, I was experimenting with the software to test its reliability when it proceeded to try to LOW LEVEL REFORMAT my BOOT DRIVE. (yes, while I was booted up on it!) It actually managed to zero the partition table and boot blocks before asking the OS to lay down a new partition, at which point the OS thankfully gave it the bird. Kudos to Disk Warrior for being able to salvage the volume, and good riddance to CharismacRAID. (aka "AnubisRAID" fyi)

--
I work for the Department of Redundancy Department.

First Mistake by fire-eyes · 2004-10-30 11:14 · Score: 1

The first mistake is Maxtor drives. Worst drives i've seen, and I am talking present time.

Yes you have five, but do you feel like replacing quite a few of them over a few years?

I run software raid1 here at home, on a 2.6 kernel, and I can say that is solid. It appears software raid in general in linux is quite solid.

--
-- Note: If you don't agree with me, don't bother replying. I won't read it.

Re:First Mistake by Hardwyred · 2004-10-30 12:27 · Score: 1

Gotta agree. I have never had a single problem with linux software raid once I understood how it worked. The maxtor drives, on the other hand, have been nothing but trouble. The issue is componded by the fact that maxtor replaces your busted drive with a remanufactored drive (perhaps all the vendors do, I've never had another vendors drive blow out while still under warranty) and of course, they fail several drives at a time.

--
www.linux-skunkworks.com
Re:First Mistake by Anonymous Coward · 2004-10-31 03:22 · Score: 0

Indeed, Maxtor drives suck so much I actively advise against them.. they die far too often compared with other manufacturers.
Re:First Mistake by Anonymous Coward · 2004-10-31 04:38 · Score: 0

I'll add my vote to Maxtor drives sucking. I've had considerably more of them go bad than any other brand. They're not even any cheaper than other same sized drives.

Re:Ok. by Predius · 2004-10-30 11:18 · Score: 1

Maxtor may not be the absolute best built drive ever, or even ev4r!, but they do have a no quibble warranty. Its nice, don't like the way the drive looks at you in the morning, call 'em up and they'll replace it right then and there. Hear a squeak, call, replaced. ATTO showing wierd numbers, call, replace. (I actually did with 2 drives, the read speed was down compared to 2 other matching models, Maxtor didn't question it and replaced the drives.)

Re:Ok. by pp · 2004-10-30 11:18 · Score: 1

Drives from just about any vendor can suck. I know people who had 1000 IBM 75GXP drives (hey, their previous models were ok!) and had to swap a drive a day due to errors (not total failures tho, just early signs of it). Although many people seem to recommend "identical drives" for raid clusters for performance reasons or whatever, mixing vendors or at least production batches does the benefit of a single vendor fuckup resulting in data loss due to all of your drives having similar early deaths.

As for controllers, lots of big installations use 3ware cards. A bit more expensive than the basic ide/sata adapters, but you get lots of ports in one pci (preferably something better than 32-bit/33MHz) slot, and each drive gets their own ide bus. Now, even if the things have hardware raid5, you might be better off using plain old software raid with them. Depending on the card model it can easily be twice as fast. Embedded CPU's aren't that fast compared to a p4 or a64 :-)

Hardware is the best by Anonymous Coward · 2004-10-30 11:18 · Score: 0

This doesn't apply to you much, but to me nothing beats ServRAID on an IBM. RAID 5 is ultra quick and I've had a few drives bite the dust(luckily never simultaneously or we'd go to tape.)

I just popped in a new one, watched the sync light blink for a while, never went offline for a second.

Good SCSI hardware RAID is the best choice. If you can't afford that (cost of a decent used car in many cases.) IBM x345 series with 2 logical drives(2 mirrored and 4 RAID 5 drives.) You can't beat it for the money.

Adaptec Software RAID on a PC server spiraled into oblivion on us once, thank god for good&strict backup strategy. It's idea of rebuild was to remove reference to files. Part of the blame should be on the tech that did it(not me) but overall it just handed him the knife and he twisted it well.

8 drives? Maybe I am missing something? by kbahey · 2004-10-30 11:20 · Score: 1

You said 8 drives? Maybe I am missing something.

You say a 500 MHz machine. That is probably a P3, and hence would not have SATA. You did not say SCSI either. If you are trying to connect 8 drives to a standard PC you will run out of IDE ports. Most PCs come with only 2 ports, i.e. maximum of 4 drives.

So, how are you going to connect them?

--
2bits.com, Inc: Drupal, WordPress, and LAMP performance tuning.

Re:8 drives? Maybe I am missing something? by Peridriga · 2004-10-30 11:30 · Score: 2, Informative

Connect them with these
Re:8 drives? Maybe I am missing something? by Anonymous Coward · 2004-10-30 13:35 · Score: 0

That's what I was wondering. Personally I'd probably go with 6 or 7 firewire enclosures and a firewire card, but that's going to set you back a few more bucks.
Re:8 drives? Maybe I am missing something? by BawbBitchen · 2004-10-30 14:12 · Score: 1

Tried that. 8 bay drive. The Linux FW driver is not up to the task. Crapped out drives on boot, dropped under heavy load. Just an FYI!

What is the definitive article? by Futurepower(R) · 2004-10-30 11:21 · Score: 2, Informative

Is this the definitive article about software RAID under Linux?

Software-RAID HOWTO. In English and HTML: Software-RAID HOWTO.

--
Bush borrows money to kill Iraqis. 140 billion borrowed. With interest, you pay 200 billion. When Saudis attack, invade Iraq?

Done and done.... by thewiz · 2004-10-30 11:21 · Score: 1

Check out the project I did over two years ago to provide my home with Network Attached Storage: http://www.mini-itx.com/projects/tuxserver/

This is an awesome little machine that has been serving files to my home network for over two years with nary a reboot. Fast, stable, and zero problems with the software RAID5. I did have one drive go out about a year ago; popped in the replacement and the array automatically rebuilt itself.

One recommendation I'd make is use a journaled file system! I use ReiserFS and it is excellent for use in a software RAID.

--
If "disco" means "I learn" in Latin, does "discothèque" mean "I learn technology"?

Another suggestion... by rnturn · 2004-10-30 11:21 · Score: 1

Invest in a UPS for this system. Any uncontrolled shutdown (i.e. power outage) will take quite a long time to fully recover from. It will take a lo-o-o-ong time for the resync to finish on an 8x250GB raidset. Then you really ought to force an fsck as well on the filesystems you've created on that monster.

--
CUR ALLOC 20195.....5804M

Software RAID on Linux by Anonymous Coward · 2004-10-30 11:23 · Score: 5, Informative

Two pieces of advice: (1) Look into mdadm, it saved my array once when I had to move it from one server to another, (2) look into smartd as a way to monitor the individual disks and detect failures. Okay, well then, _three_ pieces of advice. (3) make sure you look into ext2/3 filesystem parameters like the size of the journal (max it out) and the -R stride= option.

mdadm will allow a "spare pool" shared between multiple RAID devices and smartd will check the state of the disk controllers at regular intervals. You should put the system _and_ the disks on UPS to avoid losing data in the event of a power failure (the disks need to write their cache to the physical media before it evaporates). Set up something (mdadm or smartd) to email you in the event of a disk failure, or you may be running in degraded mode for quite a while before you discover it (unless you look at /proc/mdstat regularly).

All in all it seems to work fairly well if you spread the disks across multiple channels, if you have enough RAM for page (buffer) cache, and if you get reliable disks. I have a 4-disk SCSI storage box that I have in RAID 5 mode. It has been running for over two years. The server failed and I had to move it, that is when I discovered mdadm -- A LIFE (DATA) SAVER!

Re:CONFIGURE IT RIGHT!! small parts... by GigsVT · 2004-10-30 11:25 · Score: 1

Ahh, a comment with good insight. I never thought of it this way, but I'll definitely consider doing this in the future.

I was always stymied about not being able to partition a md software RAID, I didn't think about it from your angle, that it's not what you'd want anyway!

--
I've had enough abrasive sigs. Kittens are cute and fuzzy.

Mixing (software and hardware) RAID by rhino_badlands · 2004-10-30 11:25 · Score: 1

I did a quick scan of the post and didn't see anything about mixing your software and hardware RAID.

Alot of people will tell you to go with hardware as you have seen in this post, and some say stay with software. Basicly i would go with a mix this provides the best solution you can write across multiple drives and provide fool proof backup giving you the best of both worlds.

Apple is now using this in their XServe RAID http://www.apple.com/xserve/raid/.

If you do go with something along these lines i would definatly upgrade your CPU to at least 1 Ghz and 512 if not a 1GB of ram. Doing this would make a nice fileserver and you could even host a web/ftp server if you wanted access to your files off site.

--
- MOSKIE

depends on what you need by quewhatque · 2004-10-30 11:26 · Score: 1, Informative

I have run software raid on Linux and FreeBSD just in testing. They seem to work without hiccup and they're stable, although I had no extensive testing. As far as hardware or software, I'd suggest hardware if you need hot-swappability and speed. It's a waste of money if speed is not an option as software raid is just little less secure; I can only see it losing data in the event of the kernel going wrong in a very unusual way or more likely the kernel crashing in the middle of a write. Increasing the ram would definetely take a lot of stress off the kernel and increase speed, but then again, I haven't seen many situations where stored files need to be quickly accessed; I can only think of low compressed, hi-quality movies. As far as swapping drives, if you set up linux properly, the drive will reconstruct correctly (given only 1 dies). If linux dies and you have to redo the OS, the configuration is stored in each disk (partition more properly) that is in the raid so none is lost except time. One of the advantages of software raid is its flexibility, it's partition based instead of disk based. It doesnt seem like much of an advantage (and it isnt). Just make sure you don't make two partitions on the same disk part of the same raid array. My best piece of advice is: make sure every drive gets its own IDE controller. A drive can slow another one down on standard ATA. Also, a drive going wrong could kill the rest of the drives on its channel.

Experience by Libor+Vanek · 2004-10-30 11:26 · Score: 2, Informative

As someone who built and installed ~400 systems with about 50 TB of storage capacity and ALL on Linux SW RAID5 I can only recommend it. I have bad experiences with HW RAID - when 2 or more disks fails, you can't get your data. Linux SW RAID is easy (OK, not SO easy) to be convinced to recover most of your data except the really bad ones. Also performance is really superb (with P4/Xeon/Opteron CPUs it's much higher then any HW IDE RAID can do).

Re:Experience by Vaughn+Anderson · 2004-10-30 14:26 · Score: 1

I read this story just out of curiosity. I have never used Linux but I am willing to put in the effort to get a box going for the sake of a good storage system. Is there any tutorials you would recommend online for setting up a system like this guy has and doing it right?

Thanks!

-v
Re:Experience by flsquirrel · 2004-10-30 16:13 · Score: 5, Informative

I got a newsflash for you. If you're using raid 5 and you drop more than one drive at the same time, you're done period. It doesn't matter if you're running hardware or software raid. It's the way raid 5 works. RAID 5 can recreate the info from any one drive. But if it loses two drives, it can't determine both variables. There is no convicing about it. If two drives die, so does your array. Do not pass go. Do not collet $200.

It scares me that they let people like you play with the sort of computing resources that have 50TB of disk space.
Re:Experience by Libor+Vanek · 2004-10-30 22:17 · Score: 1

Jesus christ - if 2 disks TOTALY fails, your TOTALY done. But most (90%) of the time fails ONLY several blocks (few kB) and if it's not something critical (superblock etc.) you can recover your data EXCEPT the files which are locaten on failed blocks. And if disk TOTALY fails, usually it's only the electronic board, which you can "borrow" from same-but-working drive, recover you data and trash the bad disk.
Re:Experience by Libor+Vanek · 2004-10-30 22:22 · Score: 1

SW RAID howto.
Re:Experience by imsabbel · 2004-10-31 01:35 · Score: 1

Hm... My controller just points some new messages in the system log and remaps the defect sectors automatically.
Failure of "a few blocks" isnt a disk loss.

--
HI O WISE PRINCE. WHT TOOK U SO DAM LONG?
Re:Experience by Anonymous Coward · 2004-10-31 07:36 · Score: 0

You're now talking semantics and shades of grey. In a very black and white sense, if you drop 2 disks, you're done, period.

Now, as to borrowing electronics and such in order to recover, firstly, why? If you're talking about something that is mission critical (hence on an array), then replace the dropped disks, fire up the server, reload from last backup, sure you're going to lose data since last backup, but that's a hell of a lot better than losing hours or days due to the server being down.
Re:Experience by Libor+Vanek · 2004-10-31 10:27 · Score: 1

Depends... my experiences are if there are "few bad blocks" that means that there is something VERY bad with HDD and it should be IMMEDIATLY removed. Or are you talking about S.M.A.R.T. value "Reallocated event count"?
Re:Experience by imsabbel · 2004-10-31 23:36 · Score: 1

or, no. I meant bad sectors.
I agree that its mostly a sign of a hd shortly before dead, but normaly it has days/weeks left, MUCH more than enough to do a rebuild even of a 1TB array...

--
HI O WISE PRINCE. WHT TOOK U SO DAM LONG?

I have a 500Mhz machine by ericdano · 2004-10-30 11:27 · Score: 1

That I put a RAID5 controller into. It works great. Spend the extra $$$ on a hardware controller. It's well worth it, especially when one of the drives decides to die.......

--
It's either on the beat or off the beat, it's that easy.
I moderate therefore I rule!
--

Re:Ok. by Anonymous Coward · 2004-10-30 11:28 · Score: 0

Just for the record, english speaking countries != the world. As for my personal experience, my oldest 40G maxtor still ticks away nicely after years of +10h daily use. I have managed to blow a few seagates and western digitals though, even before their warranty had expired. Samsung's 7200rpm models seem reliable aswell.

A couple of recommendations from a Slackware user by Derf_X · 2004-10-30 11:28 · Score: 1

I have a software RAID 1 array with 2x 30 GB on a slackware 10 server. It's not as big, but it works quite well.

First the setup: it is a Pentium 233 MMX (P1, not P2) with 64 MB of ram. It was first setup with Slack 9.1 a year ago. I did a clean install when Slack 10 came out and all Slack kernels have software RAID enabled by default, so it was recognised automatically. The two drives are quite different: one Maxtor 7200 rpm drive and a WD 5400 rpm drive that have ~ 1 GB difference in size. This computer does not even have a monitor connected to it.

Now the recommendations:
-Read ALL How-To's related to RAID, you can all get them at http://www.linuxdoc.org/ and either print them or have them quickly available on another computer.
-Be sure you have raidtools installed, it's a standard slack package.
-All partitions have to be the exact same size and of the "Linux raid autodetect" type. The software RAID on Linux is partition based, not disk based, which makes it more flexible.
-All drives have to be Master on an IDE channel, no slave drives (or else it will very slow), so 8 drives = 8 IDE channels = 3 additional controllers (or 2 if you have an onboard RAID controller). At this price, it could be interresting to get a 3ware hardware based raid card.
-NEVER, EVER run a hardware RAID array on cheap Promise RAID controllers (but software RAID on regular IDE Promise controllers, OK).

As for drive failure: once, when moving my server, a power cable got unplugged from the master drive on the first IDE port, the server booted without any problem from the second drive which is master on the secondary IDE port. The CDROM drive which is slave on the first IDE port was even detected correctly. I noticed it when I did the lsraid command. I then shutdown the server, plugged the cable back and did the raidhotadd command to rebuild the drive, a little less than an hour later, everything was ok.

It's pretty much what I can think of now...

Re:Bottleneck is not CPU (it's networking) by sakusha · 2004-10-30 11:29 · Score: 1

You're all barking up the wrong tree. A high-speed RAID isn't going to be much good as a file server, the bottleneck is in the networking. Are you going to use gigabit fiber to connect to the server?

Re:One word by hugo_pt · 2004-10-30 11:29 · Score: 1

don't = do not = two words.

Quick note about RAID5 by m33p · 2004-10-30 11:30 · Score: 1

One quick note about RAID5 that not many people are aware of... If you lose *one* disk, you can recover everything. If you lose *two* disks, you had better have backup tapes, because that's the only way you can recover.

Given the cheap cost of disk these days, RAID 0+1 / RAID 10 is something you really ought to at least consider.

-p.

Whatever you do, have a GREAT PSU by Anonymous Coward · 2004-10-30 11:32 · Score: 0

You must support the platform with the best power supply you can afford. I can't stress how important this is. You don't have to take my word for it, but do the research and think about it logically. Whatever you decide, that is the most important. If you don't want to see errors accross all of the drives, get the stable PSU.

500 MHz? by tji · 2004-10-30 11:32 · Score: 2, Informative

You may not need much CPU performance for file service.. after all, it's mainly just doing DMA to/from disks. But, I assume it's just your standard PC motherboard, with a single 32bit 33MHz PCI bus.

If you're spending $960 for the disks at Fry's, why not spend another $80 to $250 at that same Fry's and get a current generation motherboard and CPU (they have package deals that are dirt cheap).

For $80, you can get a 5x faster processor, and a much newer chipset with ATA133 and Serial ATA.

For $250, you can get a board with multiple PCI busses, PCI-X and a chipset capable of handling much more throughput than a cheap PC motherboard.

The I/O bandwidth will be your bottleneck with an 8 drive RAID array. The standard 32bit / 33MHz PCI bus only does about 1Gbps. Serving a gigabit ethernet connection will use all your bandwidth by itself.. when you have 8 ATA drives fighting the NIC for bandwidth, you can see a clear problem.

If you're spending that much for the drives, don't hamstring it by skimping on the motherboard. And, in any case, once you have a Linux box installed, you inevitably start using it for many tasks (caching proxy, mail server, ftp server, dns server, www server, etc). So, a beefier system will stand up better.

Re:500 MHz? by Xerp · 2004-10-30 11:52 · Score: 1

I couldn't agree more. Spend that $50 and get a hardware RAID controller; especially with a 500Mhz processor! Don't get me wrong; I've used Linux software RAID in the past and it performed very well. I didn't have the luxury of being able to spend any money though. Everything I did was from "scrap".

And don't forget, just because you have RAID it doesn't mean you don't have to back up your data!

You also don't say what sort of drives or motherboard you have. Most motherboards have 2 IDE controllers (4 devices). Some of the newer boards also have SATA, but you're not going to have 8 ports there either. Heck, if you get a decent mobo it will have a RAID controller built in. Of course, a decent mobo wouldn't support of 500Mhz CPU ;-)

Another issue is the power and requirements; I trust you have sufficient power and cooling?

From the limited information available, I see thing you can do with these 8 drives is to split them across 2 machines, each with 4 drives in, identical configs. Configure the Software RAID-5 across 3 drives, with 1 hot spare. One machine can be used to backup the other machine.

Or, you can backup everything to floppy disk
Re:500 MHz? by Anonymous Coward · 2004-10-30 12:21 · Score: 0

Um, a $50 "hardware" RAID controller is going to be one of these POS cards that are really hybrid hardware/software controllers. They represent the worst of both worlds, and should be avoided. Go full software unless you are willing to fork out the $$$ for a real controller.
Re:500 MHz? by Jeff+DeMaagd · 2004-10-30 14:26 · Score: 1

Very few $250 boards have multiple PCI busses or PCI-X. I found a Tyan Tiger i7501(S2723GNN) at Newegg for $277 and it is a Xeon board. There are other boards that cost $450 and up that have PCI-X are Xeon and Opteron - neither chip is cheap, unless you want to buy the slowest, a 1.8GHz Xeon is $90 on Pricewatch, Opteron 140 is $150, Opteron 240 (2CPU version) is $190.

Do you mean PCIe?

A lot of boards that have on-board ethernet bypass the PCI buss because many chipsets now have a direct connection for network chips.
Re:500 MHz? by tji · 2004-10-30 15:46 · Score: 1

Yeah, you're right.. PCI-X boards are usually not very cheap. But, a few searches will turn up some good candidates that are being closed out. Any of which would blow the doors off a 500Mhz PC with 32bit/33Mhz PCI.

A search at Yahoo shopping came back with a SUPERMICRO P4DPE-Q Dual Xeon board for $247. Or, to keep the total cost down, go with a Supermicro P4SCi Single processor P4 board with dual Gig-E ports for $216.. throw in a $50 Celeron, or splurge for a $90 P4 for a larger cache.

Both of those are excellent bargains.. Buy the cheapest CPU you can find for each, and you still have a system orders of magnitude faster than the original poster's 500Mhz PC.

The bandwidth of these server systems is really great compared to a standard PC. I've done some network throughput testing on the full range of boards, and for any serious server application high performance I/O is a must-have.
Re:500 MHz? by Xerp · 2004-10-31 01:12 · Score: 1

Um, a quick search shows a wide selection.

Software RAID is probably ok for you by bicatu · 2004-10-30 11:33 · Score: 5, Insightful

Hi,

The scenario you've mentioned is probably OK to use a software RAID. I use it in a production enviroment without problem with a higher stress that your setup will probably have.

I'd suggest you to consider the following items :
a) cooling system - those HD can generate a lot of heat. Buy a full tower case and add those HD coolers to make sure your HDs stay cool

b) Buy the HDs from different brands and stores - RAID5 (either hardware or software) can recover from one drive. If you buy all from the same brand/store chances are that you end up with 2+ drives with the same defective hardware

c) cpu - if you are going to use this number of drives the processor will be a majo bottleneck. Do not forget that RAID5 XOR your data to calculate the parity.

d) partition scheme - use smaller partitions and group them together using LVM. This you help you to recover from a smaller problem without taking a lot of time to reebuild the array

Re:Software RAID is probably ok for you by Alan+Cox · 2004-10-30 12:40 · Score: 2, Insightful

Let me strongly second (b). I've made that mistake the painful way - two drives with serial numbers 5 apart did indeed die about the same time.

Be aware 2.6 LVM still seems to have some 2Tb limits.
Re:Software RAID is probably ok for you by k12linux · 2004-10-30 17:58 · Score: 1

I'll third recommendation b)
I'm not sure I would go so far as to get different brands, but I would definately try to get them from different batches if possible. If there was a manufacturing problem you'd hate to get 5 out of a batch of 12 defective drives. Multiple drive failures at one time in your RAID array is painful.
Re:Software RAID is probably ok for you by jovlinger · 2004-11-10 08:49 · Score: 1

Curious what 1) sort of death and 2) what timeframe:

1) catastrophic failure that took out the whole controller, or just frequent errors?

2) Minutes, days, weeks?

I have four identical drives in raid 5: I'm willing to turn off the machine until replacements come, but will need them to be alive long enough to rebuild.

Re:Bottleneck is not CPU (it's networking) by mortonda · 2004-10-30 11:33 · Score: 1

Ok, network could be a bottleneck, if the disk IO gets high enough. I think, though, if this is a 500Mhz system, it is likely to need more attention on the PCI/ATA bus than on the network.

Both network and IO are to be worried about before CPU. :)

Promise is SHRAID, not RAID by ErikTheRed · 2004-10-30 11:35 · Score: 3, Informative

The Promise controllers are SHRAID, which is my own non-standard acronym for Software w/ Hardware-assist RAID or SHitty RAID in less polite company. And the "promise" of true redundancy is a charade (rim-shot, please). Basically, you have all of the disadvantages of software RAID - the need to manually configure bootability of both drives (assuming you're running RAID 1 or RAID 0+1 - if you're running RAID5 or JBOD it's an even bigger pain), plus the need to have specialized drivers on the OS, etc. These controllers (Promise, Highpoint, etc.) should be avoided like the plague for technical reasons alone.

Good, relatively inexpensive IDE and SATA RAID can be had with 3Ware Controllers. 2-drive models start around $140, and they support up to 12 drives on their more expensive controllers. The drives appear as a single physical device to the O/S, whether it's Windoze, Linux, BSD, DOS 3.1, etc.

--

Help save the critically endangered Blue Iguana

Re:Promise is SHRAID, not RAID by atrus · 2004-10-30 11:52 · Score: 1

The 3ware controllers are nice. They are not the fastest by most measurements, but then you're getting very good I/O speeds with almost no CPU useage (vs. faster speeds with 60-70% CPU for "SHRAID"). I have 4 Seagate 160GB SATA drives on a 9500S-4LP. 130MB/s reads, 70-80MB/s writes, and redundancy.
Re:Promise is SHRAID, not RAID by Fallen_Knight · 2004-10-30 21:42 · Score: 1

Have you used the 3ware cards before? They are what i'm looking at for a 8driver raid 5 and i want to make sure they are reliable.
Re:Promise is SHRAID, not RAID by danpritts · 2004-10-31 03:32 · Score: 1

I can't vouch for their reliability but Promise now makes some true hardware RAID cards in addition to their "SHRAID" stuff.

Nothing wrong with software Raid in some cases by bogie · 2004-10-30 11:35 · Score: 1

Of course for several disks or Raid 5 your better with off with hardware. But for Raid 1 aka mirroring there is nothing wrong with software and I've been using it for many years at home and at small clients with no problems to speak of.

"It's a poor solution for raid, since if the OS goes, there goes your raid"

That's only a problem if you don't back up your data though. I agree use hardware whenever its possible but in a pinch software works great.

--
If you wanna get rich, you know that payback is a bitch

backup by Anonymous Coward · 2004-10-30 11:37 · Score: 0

What are you doing for backup? Assuming that these are important files that you wish to protect I suspect you also need to protect against user error which is just as likely (if not more so) than hardware error. For me, availability is less important than recoverability, so I switched from using software raid (albeit raid 1) to using primary and secondary disk devices that are 'backed-up' using rsync. I do not bother using compressed backup archives as the majority of my data are media files which are compressed anyway.

RAID 3, 4 and 5 aren't the answer by David+Jericho · 2004-10-30 11:37 · Score: 1

To clarify, if you care about data integrity rather than just disk array uptime, RAID levels 3, 4, 5 and now 6 aren't the answer. Examine RAID 1+0 much more closely.

The short and sweet of it is that on event of a rebuild, if a single drive serves up a corrupted block, the rebuilt data will be corrupt and with no way of affirming that. Google for ACNC's website, as well as BAARF. The original BAARF site seems to be down.

Disk is cheap, as are controllers.

Battle against any RAID FIVE FOUR and FREE (BAARF) by sean9 · 2004-10-30 11:38 · Score: 1

Here's the BAARF Site Cached on Google. The real site seems to be down. http://64.233.161.104/search?q=cache:8HJz0cFQHgoJ: www.miracleas.dk/BAARF/BAARF2.html+baarf&hl=en&cli ent=firefox-a Here's an excellent duscussion of RAID 5 vs RAID 10 by Art Kagel. I would definitely warn against RAID 5, even hardware RAID 5, unless you're doing 95% or more reads. The performance hit is horrible, even worse for software raid, and why? To save disks? You already have 8! Build a 8-disk RAID10 (or 6 disk with two hotspares) and be done with it. You'll never use all that storage anyway. Performance will be phenomenally better for all sorts of applications that can use the storage. If you're going to go with a card like Promise, be sure Linux includes the driver in the kernel. I had a lot of trouble with a Promise card because the driver was only available on the Promise site for certain versions of Redhat, so I was stuck and couldn't upgrade (or was afraid to). That can be a real limiting factor. Here's the BAARF Site Cached on Google. The real site seems to be down. http://64.233.161.104/search?q=cache:8HJz0cFQHgoJ: www.miracleas.dk/BAARF/BAARF2.html+baarf&hl=en&cli ent=firefox-a Here's an excellent discussion of RAID 5 vs RAID 10 by Art Kagel: http://64.233.161.104/search?q=cache:oat67e4ciCMJ: www.miracleas.dk/BAARF/RAID5_versus_RAID10.txt

Medical Imaging server by spectasaurus · 2004-10-30 11:39 · Score: 1

I've recently made a server for archiving medical images using Redhat 9 with 5 SCSI drives in RAID 5. Works great at a fraction of the price of a vendor solution (ie, $30,000 vs $1,500.) In this case, the vendor was using a Windows NT box yet. Performance suffered big time. The Linux box I use is a 2.4 GHz Celeron with 1GB RAM. The bottleneck is always in the network interface (100 Mbs) so the Celeron is more than adequate. It's been up almost 1 year already with no reboots.

Re:Medical Imaging server by yppiz · 2004-10-30 12:05 · Score: 1

The parent post makes a good point. Most of the SW / HW RAID arguments in this discussion focus on write performance. If the data you are writing is coming from a 100Mb/s ethernet connection, then that connection will saturate before your RAID system does.

--Pat

Poof-Poof-Poof by Anonymous Coward · 2004-10-30 11:39 · Score: 0

> I am concerned with reliability. How stable is the current RAID 5 support in Linux? How hard is it to rebuild an array? How well does the hot spare work? Will it rebuild using the spare automatically if it detects a drive has failed?
Poof...
I find your lack of faith disturbing... Poof...

Re:Battle against any RAID FIVE FOUR and FREE (BAA by sean9 · 2004-10-30 11:42 · Score: 1

Sorry about the bad formatting above. I thought I had it straight. First posting to Slashdot, I should have used the preview button, arg...

Worked for me in production by mikej · 2004-10-30 11:44 · Score: 3, Informative

To answer your actual question, whether or not the linux kernel's software RAID implementation is safe... "yes". I used it in production for NFS fileservers as far back as the 2.2 series; it performed wonderfully under high load then and has worked just as well when I've used it off and on since, both in production and on test systems. There are lots of suggestions elsewhere in the thread about things to avoid - multiple devices on the same IDE channel is the big gotcha: don't do it, its performance is particularly horrific during array reconstruction, just when you need it to run as fast as it possibly can. Keep those suggestions in mind when you build the system, but you can categorize the RAID implementation itself as more than sufficiently reliable.

--
Ideology breeds Hypocrisy. Just how much is up to you.

Done RAID-1 for years by jsebrech · 2004-10-30 11:46 · Score: 1

I've been running a software RAID-1 setup for years, on a 2.2 kernel (still haven't bothered to upgrade the kernel). It's been generally stable and fast. The one time I got into trouble was when I tried to compile in journaling filesystem support, which at the time wasn't compatible. As with most things in the kernel, the issues are well known, and as long as you read up on them before you'll do fine.

I do remember it being a bitch to set up though, especially so I could boot off either drive (a requirement I had for quick recovery from primary disk failure). Maybe the process has gotten easier, I haven't read up on it.

Another source of true hardware RAID by ErikTheRed · 2004-10-30 11:47 · Score: 5, Informative

This is slightly off-topic because it won't take care of the particular solution being sought, but another interesting way to do RAID-1 is using the controllers from Arco Data Protection. The have some that are physically connected between your IDE or SATA controller and the two drives to be mirrored - they just seamlessly mimic a single IDE device. This makes it possible to RAID-1 any IDE or SATA drive under any operating system or device. I've used them in places like phone systems and voice mail systems that have no provisioning whatsoever for RAID. It can take a little bit of case tweaking, and you have to be sure the power supply can handle it, but it's an interesting solution in certain situations where nothing else can do the job.

--

Help save the critically endangered Blue Iguana

Re:Another source of true hardware RAID by tarball · 2004-10-30 13:19 · Score: 2, Interesting

BTDT.

The gentleman is correct. I've used Arco in 2 systems that ran for years flawlessly. Except for a drive failure. Which made the peizo alarm become annoying and LEDs change state.

And thanks so much for bring it up, because you reminded me I had one of those tucked away, forgotten, brand new in the box. I will be putting it into service soon.

tom

I hate sigs, and refuse to have one.

--
I hate sigs, and refuse to have one.
Re:Another source of true hardware RAID by tyen · 2004-10-31 02:07 · Score: 1

Hmm...the specs say they only support up to ATA 100. So that means we're stuck with the 138 GB limit? Looking at pricewatch, it appears that the sweet spot for IDE drives in $/GB terms are 160 GB drives, and that sweet spot tends to move about 12-18 months to the next tier of capacity. I would be all over Arco Data's products if they could support the larger drives that require ATA 133 (have a RAID 5 now, but looking into building large RAID 1 caches tied together with LVM for a hierarchical storage management system because a lot of my data is read less and less frequently over time).
Re:Another source of true hardware RAID by smatthew · 2004-11-03 02:09 · Score: 1

FUD - don't listen to this guy. ATA133 drives work just as well under an ATA100 interface. The 138GB limit (if applicable) is imposed by your bios, and both windows and linux have workarounds.

Besides - if you actually looked around you'd see that their HW supports Drives up to 144 PetaBytes. So i thnk you little 160GB or 250GB drives will work just fine

See more specs here

They also have a SATA version

--
slashdot username - at - email.domain.name
Re:Another source of true hardware RAID by tyen · 2004-11-10 10:29 · Score: 1

Thanks, I stand corrected; some further research after I posted that pointed out the same thing you did, but I didn't think anyone would read it since it was scored so low. As I understand it, this means that the ATA100 interface only means lower transfer speeds than what I'm used to with AT133 drives and interfaces. If so, then I'm definitely looking into these products.

MDADM by Lour · 2004-10-30 11:51 · Score: 1

Use MDADM for building and maintaining the RAID, it is far superior to other tools for this that I have used. It will make life easier.
Second if you have a few spare buck get a 3Ware card and go hardware raid (its much faster and worth it).

--
-Lord Shadow

Software RAID Experinces by rimu+guy · 2004-10-30 11:51 · Score: 5, Informative

I manage a lot of servers remotely. I started out using the hardware RAID support on my server's mobos. But there were issues with that.

First, it was hard getting Linux driver support (I think drivers were available, but it was a matter of downloading them. And I don't beleive they worked on the 2.6 kernel's I used).

Then the RAID setup required BIOS settings. When you only have remote access to a server (and no KVM-o-IP) that means you need to work through a tech at the DC. Not, umm, ideal.

And finally, there was the issue of 'what if I need to move these disks to a different server'. One that doesn't have the same raid controller. Well, it wouldn't work.

Anyway, I ended up using software raid. I've used it now on a few dozen servers. And I'm really happy with it. Performance seems fine, albeit I'm not using it in really IO critical environments like a dedicated database server. In in 99% of cases I'd now use software raid in preference to hardware raid.

What follows are a few tips I'd like to pass along that may be a help with getting a software raid setup...

If you get the chance setup RAID on / and /boot via your OS installer (on a new system). Doing it afterwards is a real pain.

Build RAID support and RAID1,and RAID5 into the kernel (not as modules). You'll need that if you boot from a raid1 boot partition. Note: if you are using RAID5 you'll need RAID1 built in (since I beleive in the event of a failed disk the raid personaility swaps from RAID5 to RAID1).

With a 2.6 kernel build I've been getting "no raid1 module" errors at the make install phase when building with a RAID-ed / or /boot. The 'fix' is to compile the RAID support you need into the kernel (not as modules) then run: /sbin/mkinitrd -f /boot/initrd-2.6.8.1.img 2.6.8.1 --omit-raid-modules (substituting your kernel image name/version).

Every now and then I've had the kernel spit a drive out a raid array. I've found that sometimes the kernel may be being overly cautious. You can often raidhotremove then raidhotadd it back again. And you may never see a problem again. If you do, it probably really is time to replace the disk.

Rebuilding a RAID array goes smoothly. It happens in the background when the Linux machine is in multi user mode. The md code rebuild guarantees a minimum rebuild rate. From memory it takes about an hour or two to do a 200GB RAID1 array.

You can see the RAID rebuild status in /proc/mdstat. I run a very simple script to check the RAID status each day and send out an email if it is broken.

If you are using a RAID-ed /boot, grab the latest lilo since IIRC it has better RAID support than what is in the distros I use.

Hard drive-wise I've been happy with Seagate Barracudas. I've had to replace a few failed Western Digital drives. (Just my recommendation from experience, it could just have been good/bad luck on my part).

One neat trick with Software raid is that your drives don't have to be the same size. You do RAID on partitions. And your raid array sizes itself according to the smallest common denominator in the array.

Tip: always create a bit of spare space on any device you are RAID-ing. e.g. a 4GB swap partition. Then if you have a drive fail and it needs to be replaced, and your replacement varies in size slightly you'll still be able to use it. Not all 40/120/200GB drives are created with equal sizes :).

In summary: Software RAID=good. Decent performance. I've had no real kernel bugs with it. No need for BIOS access. Easy to move drives between servers. Easy to monitor failures. Non-intrusive/minimal downtime when recovering a failed devi

Comment removed by account_deleted · 2004-10-30 11:52 · Score: 1

Comment removed based on user account deletion

software raid5 under Linux by Cynbe · 2004-10-30 11:52 · Score: 2, Informative

I converted nearly our whole house network to software raid at the start of this year: Big RAID5s for our fileserver, backupserver, netserver and videoserver, and smaller RAID2s for our firewall and workstation boxes.

Overall, I'm very happy with it -- no more rebuilding from scratch every time a boot disk blows!! :) I'd started converting to SCSI boot disks everywhere, but a pair of software RAID2 IDE drives gives me a much better sense of security. My workstation did in fact blow a boot drive a month ago, and rather than being an emergency, I just ambled into Fry's after a week or two and bought a replacement and rebuilt the raid. No muss, no fuss. Feels like living in the third millenium!

I did learn various things the hard way that the HOWTOs don't warn of.

Note that you can't boot off RAID5, only RAID2. The hack they mention of putting /boot on RAID2 and everything else on RAID5 is not worth it with today's drive sizes. Give yourself a 2-16GB RAID2 with a complete bootable system on it, and save yourself mucho grief at very little proportionate cost in disk space.

As of kernel 2.4x, at least, the linux software RAID5 autorecovery is workable but less robust than one might like in the face of serious problems: I had one RAID5 setup totally destroyed because the hardware was flaky leading to constant reboots while RAID5 reconstruction was just underway. After awhile the kernel got confused about the order of the disks (which shouldn't matter, but apparently did) and the whole thing went into a Death Spiral. Lesson: If you're sure the problem is just one flaky disk, feel free to just swap in another and reboot. But if you are in any doubt, play it safe: Switch off RAID autodetect first thing (by fdisk'ing the partition type from FD back to 82. Get the hardware stable, rebuild the RAID by hand, then switch everything back to FD.

RAID5 is a comparative pain in the ass to work with vs RAID2, because under RAID2 any of the partitions can be mounted normally as a non-RAID drive in an emergency, getting you back on the air fast, but not so with RAID5. (You'll want a live Linux CD with a RAID-supporting kernel, likely. Knoppix &tc don't yet ship this way.) So only use RAID5 if the extra space really matters -- the big servers.

BTW: One of the reasons I like software RAID over hardware: If you have hardware RAID and the controller blows and you can't find a matching model, you may be stuck reverse-engineering their RAID scheme to recover your data. No worry about that under software RAID.)

I tested automatic failover to hot spare disks under the kernel, and it worked perfectly for me in a handful of tests. For whatever that's worth.

Do keep an eye on /proc/mdstats readout of your RAID system health. If you're asleep at the wheel and don't notice anything until enough disks fail to bring the whole system down, you haven't gained much. I have a crontab-driven set of Perl scripts which check all sorts of things weekly to minutely and email me if they look wrong: Checking for failed RAID drives is one of the things they do. If you don't have a comprehensive solution like that, the raidtools2 package has an ad-hoc solution specifically to email you on drive failure. USE IT.

FWIW, here's the system I've evolved for partitioning disks in such systems:

First partition: One cylinder (the innermost one): Ext FS containing a THIS_DISK file in which I record when and why I bought the drive and any interesting history it has had. In an emergency when you're suddenly shuffling eight hot drives plus a couple spares plus a dead and replacement motherboard &tc, you WILL lose track of which disk was doing what. This little partition will save you a lot of grief.
Second partition: Swap, in the outermost (lowest numbered) cylinders -- because these give the fastest transfer rates, up to about a 50% advantage. Putting swap on every disk lets the kernel stripe swaps across

Software raid is more flexible by pensivepuppy · 2004-10-30 11:53 · Score: 2, Interesting

I use linux software raid5 on seven 200G disks with LVM on top, and have had good results. It's usually going to be much more flexible than hardware raid. If I run out ide buses, I can use firewire, or scsi, or sata or whatever I want. You can also use different sizes of disks (within some limits). With hardware raid, you're stuck with the number and type of ports on that raid card, and that's it.

I make lots of smaller raid5 /dev/md partitions, then concatenate them together with LVM into one large partition. When I go to add disks, this allows me to pull one md partition out (if I resize the fs down far enough), and expand that md onto another disk, then add it back into the volume. Raidreconf still doesn't sound reliable enough, so I avoid that for resizing

If you do go wtih hardware raid, make *sure* you can do a resize of the raid without losing your data. A lot of cheap raid controllers don't allow this - you have to wipe out all your data in order to add another disk to your raid, which is usually impractical. And you have to assume you're going to expand it.

Also, make sure to turn off write caching on your drives. It's much slower, but write caching is dangerous, especially in raid configurations.

Re:Software raid is more flexible by CAIMLAS · 2004-10-30 15:04 · Score: 1

That's another cool thing about linux sw raid: you can RAID damned near every storage medium. It's entirely reasonable to have a RAID1 array locally, and then have that RAID1'd to another server - and that, is cool. I don't doubt you could probably RAID ftape drives, if you spent enough time at it.

--
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers

Lots of experience...all good by mprinkey · 2004-10-30 11:53 · Score: 3, Informative

I have build at least two dozen software RAID5 boxes over the past few years. Usually Promise controllers, Maxtor drives. Performance is generally pretty good. Here are bonnie numbers for my 1.2 TB media server (five Maxtor 300 GB drives in Software RAID5). These numbers are a little slower then other systems because it uses an Athlon motherboard. I have found that Intel chipset boards generally give read performance ~100-140 MB/sec.
[root@media root]# more bonnie20.log Bonnie 1.2: File '/raid/Bonnie.27772', size: 2097152000, volumes: 10 Writing with putc()... done: 14517 kB/s 83.2 %CPU Rewriting... done: 25060 kB/s 17.1 %CPU Writing intelligently... done: 41987 kB/s 29.5 %CPU Reading with getc()... done: 18830 kB/s 96.1 %CPU Reading intelligently... done: 82754 kB/s 62.2 %CPU
Using an older processor/motherboard is probably not a huge concern. I've used 300 MHz Celerons before. Of course, your performance might not be as high as this, but if you are using this as network attached storage (NFS or SMB), you will likely be limited to 12 MB/sec due to fast ethernet. If you have (and need) gigabit transfer speeds, you should probably use a better motherboard/CPU.

Lastly, remember that you shouldn't skimp on power supplies and an UPS that automatically shuts the system down. The *only* data loss I have ever had on raid5 arrays came because of power-related issues. Heed my warning! 8)

Re:Lots of experience...all good by mprinkey · 2004-10-30 12:03 · Score: 2, Informative

Sorry to reply to my own post. More information...avoid putting master and slave on the same port. Sometimes, if one of the drives goes, it will whack the entire port and drop out the other drive. In raid5, this is bad though unrecoverable. It might require you to manually rebuild (mkraid --secret-option) to get the data back after replacing the drive. That is a scarely situation that can be easily avoided by only using one drive per ide port.

That information may be (and probably is) outdated with regard to SATA. I don't have experience with them yet, though I will be building four 1.75 TB RAID5 (or 1.5 TB RAID6...Linux 2.6 willing) arrays next month that use 250 GB SATA drives.

thanks (was Re:Stick with hardware RAID) by Anonymous Coward · 2004-10-30 11:56 · Score: 0

Your informative reply makes perfect sense now that you explain it. Thanks for taking the time to explain, it's much appreciated.

It works well - BADBLOCKS WARNING by dbullock · 2004-10-30 12:01 · Score: 1

"I am trying to build a large home drive array on the cheap. I have 8 Maxtor 250G Hard Drives that I got at Fry's Electronics for $120 apiece. I have an old 500Mhz machine that I can re-purpose to sit in the corner and serve files."

I just setup a very similar system on an Asus A7N8XE Deluxe nForce 2 motherboard. I have 6 200GB PATA IDE hard disks, 2 on the mainboard IDE, and 2 each on Promise Ultra133TX2 IDE cards, 1GB RAM, and a Duron 1800MHz CPU.

The board rocks. Onboard USB2, Firewire, SATA, PATA, Gigabit Ethernet + 100MB Ethernet, and dual banked DDR RAM for extra speed. It runs just fine with a stock kernel and no binary drivers.

Software RAID is fine for personal systems (which mine is). I'd use an Escalade or similar card for a business system. It simply wasn't in the home budget which is prioritized for the wife, 2 year old and 3 month old instead of leet rigs.

Unfortunately I have no idea how hard software RAID is to setup on a Slackware type box. I switched over to Red Hat in the mid 90's when other things were drawing more of my time and interest than maintaining packages. Fedora Core 2 has a point and click RAID creation facility during installation but even as a GUI tool it's real drudge work to setup (unless you just make 1 giant partition)

A few lessons I learned:

*** RUN BADBLOCKS ON YOUR DISKS BEFORE USING ***

I skipped doing this and lost a few hundred gigs of replaceable data that nevertheless i'm not looking forward to replacing. I know backup backup backup. Easier said than done at those volumes. You need to run badblocks in one of the write modes because IDE hard disks will not remap bad sectors unless they're written to first.

Let it run overnight. Before running badblocks one of my drives kept eating my data, and it would not pass a SMART test. I was ready to RMA it before I stumbled on the fix. After running badblocks on all drives once all night long I've been rock steady. I now run SMART tests periodically and use smartd.

Between 2.6.5 (Fedora Core 2 install kernel) and 2.6.8-1 Linux changed the order that the drive devices are enumerated, which caused weird boot problems until I figured out that it was trying to boot drives off my SATA controller that I don't have when it suddenly made the SATA /dev/hda-/dev/hdd instead of /dev/hdm-/dev/hdo.

I chose to use REISERFS. The people at namesys are great, and I don't have to worry about having 964GB of space and no inodes left.

I initially had 4 drives on my Promise card. I was less concerned about speed and more about space since this is a personal server. I had tons of interrupt errors and it turns out the Promise 133TX2 cards can't really handle 4 DMA100 drives, breaking the drives up to multiple cards made a big difference.

Incidentally - I'm running about 3GB of SWAP (I currently have 1GB of RAM - but I saw no performance difference when it had 256MB), with a 4GB root partition, and the rest is under /mnt/data1.

Heat is another HUGE problem when we have this many drives. In my initial setup I figured just having fans was enough. When my internal case temperature hit 160+ degrees I reconsidered. I now run the case open with a fan blowing across it.

- Dave

--
http://www.bullnet.com

Software RAID during recovery by wildjim · 2004-10-30 12:01 · Score: 1

I've not tried s/w RAID-5 under Linux, as, among other things, a Solaris DSO course I went on recommended against it so heavily -- and not just because Sun wanted to sell us more hardware ;-)
But anyway, I've striped partitions across Linux (LVM2) and Solaris disks (Disk Suite/Vol Manager) several times -- RAID 0 I think that is -- and that seems to be pretty efficient.

Mostly it has to do with the fact that good h/w RAID has one channel and possibly bus dedicated per disk (whether SCSI or IDE) which means there's very low access-contention, but it also helps (a lot) that the dedicated controller handles all the splitting/consolidating of the data across the disks, parity-checks when a disk is replaced, etc, etc, so the OS can get on with what its better at.
Sharing a channel/bus (e.g a single IDE cable or a single PCI Bus) can really affect read/write performance because the data will have to be consolidated in a divided manner across those same channels/buses from all the disks (or all-but-one for RAID-5). The speed of those channels/buses can vary a lot between systems and architectures, though a 500 MHz CPU suggests a less-than-desirable speed for this task...

Parity-checking a replacement disk will also absolutely destroy disk-access speed -- *all* the data on *all* the disks has to be read in to write the correct parity to the new disk -- but a good h/w RAID should still allow the array to be used at the same time without too much pain.

Anyway, since you've got the disks already, it would be worthwhile playing with RAID and finding out for yourself, but if you can can't spare the time (or headaches) I'd definitely think seriously about a h/w RAID solution.

Do you really need RAID? by gurgi · 2004-10-30 12:03 · Score: 1

If you are storing data that will not change much, it may be simpler to simply make a cron job to copy the data from one drive to another. Set the copy to happen in the middle of the night and you're good. You'll have a copy of the data on a drive that you can put into any machine, and not have the overhead of software raid. I've used software raid, but found that it is slow when you have more than one drive on an IDE channel.

just for kicks, check out vinum by Anonymous Coward · 2004-10-30 12:03 · Score: 0

vinum under freebsd. under 5.3 use gvinum. laid out like a real volume manager with volumes, plexes and subdisks. For a reasonably current cpu, s/w raid is as fast as h/w. If you wanted real perf. you'd of bought SCSI so you'd have tagged queueing. Failing that, SATA on true SATA drives/controllers (not the bridged PATA ones).

UPS. A good filtering one with a minimum of 10 minutes of ride through. UFS2 on 5.3 will also save you waiting on fscks.

Re:just for kicks, check out vinum by Anonymous Coward · 2004-10-30 12:49 · Score: 0

Isn't gvinum broken in 5.3 (corruption bug)?
Also how would you suggest partitioning a 200MB drive for RAID1, on a box that has 256MB memory? Would it be ok to do something like this:
da1a 128MB UFS2 /
da1b 512MB swap
da1e 256MB UFS2 /var
da1f 10GB UFS2 /usr
da1g (rest) UFS2 /home
Sorry if I got the device names wrong, I'm new at fbsd. ;-)
Anyway I don't know if it's possible to boot onto RAID1 root, but that doesn't matter. I'm more concerned about being able to RAID1 the /var /usr and /home partitions. And I'm really curious if it's a problem if /home is very big (200MB).

What are you using it for? by Thaidog · 2004-10-30 12:03 · Score: 1

RAID 5... for what? How is it going to be connected to your other computers? Do you have 64bit pci? Total system throughput and network latency will be other issues you'll need to account for. I've used SuSE software raid and OS X software raid... OS X is running like a champ but SuSE's array failed after about a month or so. Not sure if it was a software issue but I'm pretty sure it was. If it's for something like serving mp3s or small files it will probably work just fine but if you're going to be streaming video other than anything above mpeg-2 you should invest in a nice low-end fibre channel solution and save the cpu for something else.

--

||| I still can't believe Parkay's not butter.

Comment removed by account_deleted · 2004-10-30 12:12 · Score: 5, Informative

Comment removed based on user account deletion

performs great by eatjello · 2004-10-30 12:13 · Score: 1

I have used software RAID-5 under linux for a year or so, and it has survived two hot-swapped spares, transferring of CD and DVD images, backups and restores of entire 120GB hard drives, etc. Transfer speeds are as good as a single physical drive, especially since each drive has its own channel (this may suffer a bit if you put both a master and a slave on each channel). i have done three OS re-installs and a few kernel upgrades without losing any data. My file server has only 512MB of ram, and an athlon 1800+, so it's not superpowered or anything, and my array has a terabyte of usable space, so it should be comparable to your situation.
The only PITA part of software raid under linux is the initial array rebuild, which took 3 days (under load) to complete. However, during that time, the raid is still accessible, just not redundant.

Re:Ok. by realdpk · 2004-10-30 12:15 · Score: 2, Insightful

Everyone has their own experience with a bad hard drive or a bad batch of drives from a vendor. I know people who swear Seagate is the worst manufacturer, and then some that say WD is awful, and Maxtor, and etc etc etc.

I've used them all, Seagate primarily though (SCSI servers), and have noticed a trend. They all suck the same!

The sooner we can move to cheap solid state storage the better.

here ya go, lazy man by CAIMLAS · 2004-10-30 12:16 · Score: 1

Nothing like doing a little cursory research at the more prominent documentation sites for topics on this matter.

In summary:
How stable is the current RAID 5 support in Linux? Quite. It's really the only way to go, and performs about as well as hardware raid.

How hard is it to rebuild an array? It's not. At all.
How well does the hot spare work? Seamlessly. Be sure to use LVM, as it makes things all the more seamless.
Will it rebuild using the spare automatically if it detects a drive has failed? Yes.

--
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers

Re:here ya go, lazy man by CAIMLAS · 2004-10-30 12:22 · Score: 1

I might want to add: You want only one drive per controller! This way no more than one drive will die at a time due to non-disk related hardware failures, aside from something like the power supply blowing up.

Personally, I'd see such a setup for a small home network as overly redundant in some areas, and not redundant enough in others. A better setup, IMO, would be using a RAID1 fileserver with a mirror on a workstation. If price is no objective, I'd probably go with the workstation/secondary mirror anyway, just to take into account things like power supply breakage.

--
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers

Comment removed by account_deleted · 2004-10-30 12:19 · Score: 2, Insightful

Comment removed based on user account deletion

Re:Ok. by Large+Green+Mallard · 2004-10-30 12:19 · Score: 1

Apple uses Maxtor in their lower end Macs (and around the year 2000 used them in most of their product line).

I've replaced 5 faulty Maxtor hard drives out of recent Macs owned by friends. Ergo, imho, the drives suck and I don't want anything to do with them.

Interesting by AgentAce · 2004-10-30 12:20 · Score: 1

I was very intrigued by your sudden interest with data integrity and redundance after just buying Maxtor drives...

Look at raid 6 by nzkbuk · 2004-10-30 12:24 · Score: 1

I've used software raid 5, I've always found it more reliable than hardware. and i've recovered from multiple disk failure.

I am looking at switching to raid 6 on a 2.6 kernel. 2 disk redundancy.

AVOID Multiple Promise IDE cards for soft RAID by hirschma · 2004-10-30 12:28 · Score: 1

FYI: Promise has seeminly added a crippling feature in their latest non-RAID EIDE controllers to avoid folks making cheap soft RAIDs.

You used to be able to use as many Promise PCI IDE cards as you had PCI slots - I've had up to 5 Promise cards previously, ATA-66 vintage. However, the newer PCI IDE cards will give fatal errors if you have more than 2 in your system - drive timeouts, write errors, etc.

I confirmed this with their tech support. This is something in their BIOS, and cannot be avoided. I'd guess that they purposely did this.

This is not an issue with "generic" IDE controllers, like the one that SIIG makes that has a Silicon Image chip on it. Same price, no hassles.

Linux soft RAID 5 _super_ stable by hirschma · 2004-10-30 12:32 · Score: 1

I have two servers at home using it. One is a file server that has been in use for four years, no issues. The other is somewhat newer, bigger, and has been in use for 6 months. Built one for a friend that is over 1TB, also going strong. They do 50-60 MB/sec easily on reads.

Stick with fairly decent hardware - cheap Intel branded board, good "generic" IDE controllers. Give each drive it's own IDE port. You'll be fine.

Compaq smartarray cards are backwards compatible. by Mustang+Matt · 2004-10-30 12:35 · Score: 1

For the most part the compaq smartarray line of cards are backwards compatible and you can upgrade or downgrade the cards without changing anything.

I have some limited first hand experience with this but I have only done it a few times so maybe someone else can tell of their experiences.

--
The man who trades freedom for security does not deserve nor will he ever receive either. - Benjamin Franklin

Re:Bottleneck is not CPU (it's networking) by Erwos · 2004-10-30 12:35 · Score: 1

"the bottleneck is in the networking"

I'm not sure about that. A gigabit is still 125+ megabytes a second - it might bottleneck your burst rate, but it's unlikely your array is going to saturate it besides that. Especially if multiple people are using them at once.

If you're really concerned about bandwidth, I would buy a second NIC for all your computers and simply make a "storage network". That's something of a hack, but I think it would work pretty decently.

Of course, you're SOL with wireless.

-Erwos

--
Plausible conjecture should not be misrepresented as proof positive.

Re:hardware performance can be poor by Splork · 2004-10-30 12:37 · Score: 1

the older model 3ware cards work well but have less than stellar performance. you won't get more than 30-35mbyte/sec tops of hw raid5 thruput on a 3ware 8xxx or 7xxx controller.

that said the 3ware controllers are the -best- multiport ide controllers out there, especially for SATA support under linux.

We use them and run linux sw raid5 across all the individual disks on the controller. Excellent performance (it max's out the 32bit PCI bus, easily 100mbyte/sec reads)

I haven't tried their new 9xxx series cards. hopefully those have real speeds for their hw raid5.

awesom by Anonymous Coward · 2004-10-30 12:39 · Score: 0

Linux RAID 5 is awsome. Mainly in Whitebox 3.0 (redhat enterprise 3). We have been hosting an Oracle database on a dev machine... and will use it for a backup machine... also have an email server with raid 5 for home dirs and mailboxes. Make sure you have plenty of ram and a fast machine... but you can skip that even if superfast speed isn't the issue. Even then the performance isn't THAT bad. Please note though these are all P4s with at least a gig of ram... actually shouldn't be that hard... also they are SCSI3.

Maxtor by Anonymous Coward · 2004-10-30 12:39 · Score: 0

Look at the MTBF for a Maxtor drive. Look at a 60% duty cycle on the drives (low for a RAID that gets moderate usage). The MTBF numbers for those maxtor drives is going to suck. Really really bad. If you use it heavily, it's about a 90% duty cycle. Those maxtors will die every five minutes. Make sure you've got at least one hot spare. And if you plan on using this for anything heavy...buy some seagates.

Use NFS...duh. by jusdisgi · 2004-10-30 12:42 · Score: 4, Funny

Jeez, I've never seen so many plain fools in all my life. Hardware RAID controllers! How quaint.

Here's what you do with those 8 fine drives of yours.

You'll need 9 486's. Get some sort of *nix on each one, preferably several different Linux variants and at least 2 BSD machines (I'd say more, but you know, netcraft confirms and all....) and get them all networked together. Put one drive each in 8 of the machines, format with the filesystem that's most convenient for the system on each box, and get an NFS server going serving that partition.

Then, on the ninth box, mount all the NFS shares and software RAID them.

Trust me. This is exactly what you want to do, and anybody who says different is a dumbass. People who point out what they will invariably say are "obvious shortcomings" of this setup are merely trolls, and not worth your time reading.

--
Given a choice between free speech and free beer, most people will take the beer.

Re:Use NFS...duh. by Anonymous Coward · 2004-10-30 13:19 · Score: 0

Congrats! Probably the worst advice on slashdot of all time!
Re:Use NFS...duh. by otis+wildflower · 2004-10-30 14:18 · Score: 1

So _YOU'RE_ the guy that did that at my company...
Re:Use NFS...duh. by Anonymous Coward · 2004-11-01 07:35 · Score: 0

its called a 'cluster'.

Mod UP please! (Re:Vinum with FreeBSD) by Nick+Driver · 2004-10-30 12:44 · Score: 3, Interesting

Vinum on FreeBSD absolutely rocks! You're old 500MHz machine will run FreeBSD beautifully too.

Anybody here remember Walnut Creek's huge ftp archive at "cdrom.com" which back in it's heyday of the late 1990's used to be the biggest, most highest traffic ftp download site on the planet? They used a combination of Vinum software raid and Mylex hardware raid to handle the load. I remember reading a discussion article from them once that until you get a totally ridiculous volume of ftp sessions hammering away at ther arrays, that Vinum was actually a slight bit faster than the hardware array controller.

Your Bad Drives Are ALWAYS At Fry's by Anonymous Coward · 2004-10-30 12:45 · Score: 0

I have 8 Maxtor 250G Hard Drives that I got at Fry's Electronics for $120 apiece.

Bad idea. If they're 6Y250MO's, I bought one of those too at the same (recurring) sale price, and have had nothing but trouble with it since. Same story with 70-80% of drives I've bought at Fry's, and I've bought MANY. I know I should be heeding my own advice, but I've always been lured by the rock-bottom sale prices (hard drives are one of the few product categories where Fry's rebates are actually honored,) and by the recurring delusion that "if it's boxed and factory sealed by Maxtor/WD/Seagate, it can't be that bad..." Unfortunately, there seem to be three hard drive Q/A outcomes, especially for Maxtors: "Pass," "Fail" and "Fry's."

Re:Your Bad Drives Are ALWAYS At Fry's by ATMosby · 2004-10-30 20:57 · Score: 1

And at Best Buy.
So not too long ago there was a good sale on drives at Best Buy. Went into one store and the boxes all looked like they'd been in a Samsonite advertisement.
You know the one with the angry sysadmin that smashes everything.
Well at least it looked like one of our sysadmins! :-) Needless to say, I didn't buy any of the drives.

For 3D fun by Kludge · 2004-10-30 12:45 · Score: 1

Sometimes hardware RAID does sucks. We had a HW controller (I think it was 3ware) that sometimes took 30 sec to delete a 100M file. We sent that one back.

Software RAID does have some real advantages, like speed. Put disks on multiple SCSI or IDE controllers with a dual CPU and you can _really_ fly. Also, you can recover from some really screwed up problems with mdadm, if you know what you're doing. Consult the linux raid mailing list: the feedback is great.

That said, hardware raid is much easier.

Redhat by Anonymous Coward · 2004-10-30 12:46 · Score: 0

Just make sure that you 'mkinitrd --with=xor --with=raid5' (read man page first please) in that order... or the modules won't be there at boot and you'll have to start the raid5 some other way.... but otherwise 1gig ram and p4 should do the trick. Performance is good. Oh and you might wan't a hot spare in the mix. Finally and VERY importantly adjust the partitions down from the max size for the drives a bit... say go with 148 on a 150 gig drive. Reason being that not all drives are the same and if you ever have to replace a drive you'll be sol if the total size of that drive is smaller then the others in your raid5.

Fibre Channel and Software Raid in Linux by Anonymous Coward · 2004-10-30 12:50 · Score: 0

Perhaps someone would know if a JBOD (Just a Bunch Of Drives) was mounted on a fibre channel network, could more than one linux box access it as a Raid 5 array? In other words if the only boxes that were reading from the storage array were linux, would it work for each of the machines to see the array as a whole? Im just thinking of some interesting plans for some spare fibre channel equipment.

limitations by Anonymous Coward · 2004-10-30 12:50 · Score: 0

Most systems will not allow more than two additional 2-channel IDE controllers. It is advised to have only one device per channel. Your best option is a six or eight device raid5 controller that uses software parity.

power by Anonymous Coward · 2004-10-30 12:51 · Score: 0

Tech support was totally right. Regardless of what the power rating is on your power supply, you can't keep the power within the required limits with that much load, especially at boot. You need to use an external enclosure to keep the power from glitching or dipping at boot.

my experiences by Keruo · 2004-10-30 12:53 · Score: 1

I have machine running with software raid-5 that uses software with database(mysql) connectivity for roughly 100-300 users simultaneously, load isn't that bad, ~30 queries/second, which gives about 0.05-0.30 average system loads (2gig/512ram)
The raid is made from three SATA disks, and each drive has its own controller.

Biggest problem that I have encountered so far is that when I upgraded from 2.4 series kernel to 2.6, the sata devices changed from hdx to sdx and suddenly the arrays vanished.
But that was easy to fix just by telling the software where to look for the drives.

Recovery seems to work well from past experiences on older system. Had ide drive failing, I didn't have spare available right away so I just mounted the array as degraded readonly meanwhile.
After getting replacement, array took awhile to rebuild, and ~8 hours later it was up and running in readwrite mode again.
Haven't tried if hotplug works with serial ata drives but I'm bit suspicious and I'd rather take the system down for a moment to fix it instead.

--
There are no atheists when recovering from tape backup.

RTFM? by kir · 2004-10-30 12:55 · Score: 1

I don't mean to sound like an ass, but has this guy even RTFM?

--
3cx.org - A truly bad website.

Possible to use 2 power supplies in one case? by Futurepower(R) · 2004-10-30 12:56 · Score: 1

And keep all those drives cool; use fans.

It seems to me it should be possible to put another power supply in the case and power the extra drives from that. Most modern BIOS versions allow specifying a delay on the hard drive spin-up. Just put the power-on connector in parallel with the main power supply connector. I don't like Antec cases much, or those from any manufacturer, but some full tower Antec cases have room for a second power supply.

But a case with a lot of drives and a motherboard needs a lot of air-moving equipment.

--
15 of the 19 9/11 attackers were Saudis. Many don't like the U.S. Gov. influence on their country.

Re:Possible to use 2 power supplies in one case? by IgnorantSavage · 2004-10-30 14:27 · Score: 1

There might be grounding problems with this setup, since each power supply will have a separate ground. The grounds for the power supplies will probably end up connected through the SATA or IDE cables, which could lead to major 'ground bounce' and kill signal integrity.

If he has a multiple power supply configuration, perhaps this is his 'dirty power'. Might be solved by creating a separate very low resistance connection (perhaps a low gauge copper wire) between the grounds on each power supply. Or perhaps the dual power supply cases come with a solution to this? I'm not familiar with these.

As someone else pointed out, heat could also be a problem. Overheated components can cause all kinds of weird problems.
Re:Possible to use 2 power supplies in one case? by ErikTheRed · 2004-10-30 15:59 · Score: 1

This is very true; I've actually killed components with this phenomenon in my younger and more foolish years.

--

Help save the critically endangered Blue Iguana
Re:Possible to use 2 power supplies in one case? by Anonymous Coward · 2004-10-30 16:19 · Score: 0

if 2 psu's in a single chassis is new to you and you're job is in IT you need to quit, now, do not wait until tomorrow, just quit, go find some farmer who will let you shovel his pigs shite and stay there forever...

Wow, the ingorance and idiocy on this thread is just overwhelming. People, there are chassis out there with dozens of PSUs, grounding is well taken care of IF A REAL ENGINEER DESIGNS THE DAMN THING and as far as heat, jesus, get with the program, we've been cooling a lot hotter things for years with a little thermal ENGINEERING...oooh boy, it's that engineering thing again...I know, we're talking about a bunch of IDIOTS who can't even realise that if you're serious about your data that you buy a second controller as a backup, and that hardware RAID KILLS any software RAID and always has -- always will. I don't care how cool your system is or how you have configured your latest version of your half-ass-cracked kernel, or whatever you have done, get with the program, learn something or get the hell out of the industry.
Re:Possible to use 2 power supplies in one case? by Anonymous Coward · 2004-10-30 21:41 · Score: 0

Ah, I almost read 10 posts without encountering the elistist-asshole-prick-post. Know what, jerk-off? FOAD. You added no insight to the conversation and are probably just another clueless fuck who thinks they know everything because they overclocked their home PC.
Re:Possible to use 2 power supplies in one case? by Anonymous Coward · 2004-10-31 00:09 · Score: 0

Elitist? wow, I've never been called a democrat before...now I'm insulted...

I do have insight, a lot of it, design of 22 designed power supplies that worked with a >300000 hour MTBF within computer systems - fanless...

I just don't feel that you pricks who can't even understand the benefits of hardware RAID shouldn't be exposed to a real discussion since you overwhelmingly think that somehow your cracked and hacked version of linux is going to somehow run more reliably and faster than hardware RAID. It's wrong, flat wrong.

Instead it's more fun to anger you all and be a troll.

Maxtor 60GB drives for $18 after rebate by Futurepower(R) · 2004-10-30 13:01 · Score: 2, Interesting

Office Depot had an 18th anniversary sale, and was selling Maxtor 60GB drives for $18 after rebate. Bought three for my personal test machines, and used my friend's addresses for the rebates.

I often hear bad things about Maxtor drives, but after a whole 40 hours use, they haven't failed once.

Linux Software Raid vs. BSD by Ragica · 2004-10-30 13:02 · Score: 1

Anyone have experience and knowledge to compare these two animals?

Cooler HDs are more reliable? by Futurepower(R) · 2004-10-30 13:04 · Score: 1

We have good luck with drives. The reason appears to be that we install them with their own fans, so each is individually cooled.

--
100 Facts and 1 Opinion -- The Non-Arguable Case Against the Bush Administration

Re:Cooler HDs are more reliable? by Anonymous Coward · 2004-11-01 05:04 · Score: 0

I don't think it is the temperature. I think it is that you are keeping them at a constent temperature that is giving you good results. I actually think that having a drive to cool is a bad thing since the lubrication fluids don't work as well at lower temperatures. I like to keep mine at about 80F, not to hot, not to cold.
Re:Cooler HDs are more reliable? by tomhudson · 2004-11-04 16:17 · Score: 1

The reason appears to be that we install them with their own fans, so each is individually cooled.
Tried that on one box, one of the drives started acting flakey after a few months. Turns out the fan for that hd was no longer balanced, and was shaking the drive. Removing the fan solved the problem.

Yes, but Silicon Image? by Futurepower(R) · 2004-10-30 13:08 · Score: 1

I've had really bad experiences with early Silicon Image based controllers. Those Koutech controllers use Silicon Image chipsets. Is Silicon Image better now?

--
George W. Bush's brother was shown in a lawsuit deposition on 20/20 talking about his prostitutes and using government influence to make money. Family values?

Re:Yes, but Silicon Image? by toddestan · 2004-10-30 15:26 · Score: 1

've had really bad experiences with early Silicon Image based controllers. Those Koutech controllers use Silicon Image chipsets. Is Silicon Image better now?

At $17, I wouldn't expect a whole lot.

PSU - single point of failure!! by Anonymous Coward · 2004-10-30 13:11 · Score: 0

You should _NEVER_ forget that the power supply you hook up the drives to is your SINGLE POINT OF FAILURE.
It it breaks _ALL_ your drives are killed instantly and your data is up in smoke. It happened to me with a raid array and no need to tell you how fscked I was.. If you're lucky to have spare parts to replace the electronics on the drives you might get your data back, but other than that.. i know it's a pain in the ass to back-up a terrabyte raid array.

All this is already replied but then, again.. by tuomoks · 2004-10-30 13:14 · Score: 1

Boot/system - mirror it, forget RAID ! Application data - depends.. Most important, separate the logical and physical access thinking. Indexed data - mirror keys, RAID data! Now - I know, mirroring is (kind of) RAID but not really the same. It really depends on what is the purpose, much less what is the technology. You back up the data ? You use raw disk access or file system ? You use journaled file system or not ? If so - where are the journals ? See - there is much more than just RAID. And so on, and so on.. Have fun.

Why not hardware RAID? by Fragmented_Datagram · 2004-10-30 13:15 · Score: 0, Redundant

If you can afford hardware RAID, go with a 4-port or 8-port card from Adaptec.

External cases by Matt+Perry · 2004-10-30 13:21 · Score: 1

What type of external enclosure can one use for ATA or SATA drives? All I've seen for sale are SCSI or Firewire enclosures.

--
Slashdot: Failed Car Analogies. Amateur Lawyering. Anecdote Battles.

Linux software RAID is not ready for prime time by laing · 2004-10-30 13:23 · Score: 1

I've been running a RAID server with Linux for about 6 years. I ran software RAID-5 for the first 2 years or so and there were several things that came up. There were a few times when the system crashed and the array was out of sync (not all the drives had the same sequence number for the last write). It required some manual troubleshooting and head scratching. There was also the issue of strangeness when the hardware failed. A few times I had a drive go bad in such a way that it was not visible to the BIOS or boot scripts. In those cases the software RAID drivers got confused. They would not allow the operation of the array with a missing disk. I had to install a drive (any drive) even to just use the array in a non-redundant mode. Also, if you're going to do software RAID with IDE drives, absolutely DO NOT use more than one drive per controller channel. It may work for a short time but you WILL have problems.

After a few years of problems like this, I never had a warm feeling about the reliability of the system so I installed a 3ware hardware RAID controller (IDE). After another few years, I upgraded to another one (SATA). I've been most impressed with 3ware's product performance, reliability, and support.. I highly recommend them.

--
JSL

Buy yourself a good hardware raid card by AaronW · 2004-10-30 13:26 · Score: 3, Informative

After months of problems with DMA timeouts and lockups caused by using a Highpoint RAID controller and a Promise IDE controller I finally bit the bullet and bought a 3Ware Escalade controller. All the sudden, everything is completely stable.

Do yourself a favor and get a good hardware raid controller and make sure it has good Linux support. Promise sucks. They advertise Linux support on the box - they lie, only with specific 2.4 kernels. 3Ware has good driver support for Linux included with the Linux kernel source code.

-Aaron

--
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.

Re:Buy yourself a good hardware raid card by Ed+Random · 2004-10-31 09:24 · Score: 1

After months of problems with DMA timeouts and lockups caused by using a Highpoint RAID controller and a Promise IDE controller I finally bit the bullet and bought a 3Ware Escalade controller. All the sudden, everything is completely stable.

The horror of PCI ATA-cards... I once had 2 Promise TX-100 cards in one machine (4 disks / 4 channels).

After some unnerving rebuilds (raidhotadd) due to IRQ problems I ended up connecting the 4 drives to 2 channels - blergh! Replacing the cards with TX-133 didn't help either. Right now, I'm running the master RAID off a HighPoint dual-channel card, for lack of better hardware...

Someone, please point me to some decent multi-port (non-RAID) ATA expansion cards. I can only find Promise and Highpoint here.

Or is a 3Ware also configurable as a "8-port ATA card"?

--
-- Gxis! Ed.
Re:Buy yourself a good hardware raid card by adolf · 2004-10-31 18:05 · Score: 1

Also included in the Linux kernel source code is a very fine software RAID 5 implementation. It always works.

In fact, it works SO WELL that of all the postings here, nobody has managed to cite a single problem with it.

As an aside, I'm curious:

Was it fun rebuilding your array and reloading your data every time you switched RAID controllers, while trying to find one that actually works? (heh, heh)

Do yourself a favor: Next time, forget hardware RAID under Linux unless you've got a good reason not to. Pocket the savings (not buying a 3Ware Escalade affords a lot of beer), or spend it on more CPU and RAM.

--
Kid-proof tablet..

RAID5 is for High Availability, not Storage! by Erisian+Pope · 2004-10-30 13:27 · Score: 1

A great deal of the posts here seem to be implying that the purpose of RAID is to protect your data. This is dangerous. Most data is not lost through hardware failure, but rather through human or program error. If you accidentally do an "rm -rf" on the wrong directory, RAID will still destroy the data. Protect data with backups: backup to tape, backup to another computer, backup to DVD or CD, but never think of RAID as a substitute for backups. My other advice is keep in mind that if you don't watch your logs you may have a drive fail and not know about it for some time. Linux RAID should be paired with log monitoring of some sort, even if it's just a cron job that greps for raid messages.

Re:RAID5 is for High Availability, not Storage! by RedDirt · 2004-10-30 15:05 · Score: 2, Insightful

I have a bit of a nit to pick with your subject. =P

While I have to agree that data can be lost because of user error, I built a 2tb RAID 5 out of Maxtor 300gb SATA drives and have thus far had one in five of the drives fail. And, of course, two drives failed within a day of each other so I lost the whole shebang. RAID 5 is fine for stuff like movies and music but I'm sticking to RAID 0+1 for the really important stuff (along with good rsync backups of course).

So, "RAID5 is for High Availability but not Security" might be a better way of expressing your sentiment.

--
James
Re:RAID5 is for High Availability, not Storage! by Fallen_Knight · 2004-10-31 00:31 · Score: 1

thats what raid 50 is for:)

Get a used 3ware controller by pyite69 · 2004-10-30 13:31 · Score: 1

You can get a relatively slow 5800 for under $50, which should perform better than software RAID. For a little more, you can get a screamingly fast 64 bit PCI card.

These allow Raid 0, 1, 5, 10, and have great Linux support including a daemon to fiddle with the settings and the option of a hot swap drive bay.

RAIDFrame under OpenBSD works great! by BawbBitchen · 2004-10-30 13:32 · Score: 2, Informative

I have been using RAIDFrame under OpenBSD for about 2 years now. Never had any issues. From dmesg:

wd1 at pciide1 channel 0 drive 0:
wd1: 16-sector PIO, LBA48, 156334MB, 320173056 sectors
wd2 at pciide1 channel 0 drive 1:
wd2: 16-sector PIO, LBA48, 156334MB, 320173056 sectors
wd1(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 5
wd2(pciide1:0:1): using PIO mode 4, Ultra-DMA mode 5
pciide1: channel 1 configured to native-PCI mode
wd3 at pciide1 channel 1 drive 0:
wd3: 16-sector PIO, LBA48, 156334MB, 320173056 sectors
wd4 at pciide1 channel 1 drive 1:
wd4: 16-sector PIO, LBA48, 156334MB, 320173056 sectors
wd3(pciide1:1:0): using PIO mode 4, Ultra-DMA mode 5
wd4(pciide1:1:1): using PIO mode 4, Ultra-DMA mode 5
Kernelized RAIDframe activated
raid5 (root): (RAID Level 5) total number of sectors is 960429504 (468959 MB)

The setup is very simple.

# cat /etc/raid5.conf
START array
1 4 0

START disks /dev/wd0a /dev/wd1a /dev/wd2a /dev/wd3a

START layout
32 1 1 5

START queue
fifo 100

It has been quite stable. The box is a 1Ghz AMD with 256MB of ram.

I tried running the same setup under Linux (Gentoo & Slackware). The software RAID would crack under load and fail a disk. I really would give OpenBSD a try with RAIDFrame. You have to recompile the default kernel with RAID suppot but under BSD it is very simple. CVS the source down and:

cd /usr/src/sys/arch/$ARCH/conf
cp GENERIC RAID

vi RAID and add:

pseudo-device raid 9 # RAIDframe disk driver (Make the number, 9 in this case one more then the number of disc you are using)

option RAID_AUTOCONFIG

save the file and:

config RAID

cd ../complie/RAID
make clean; make depend; make
cp /bsd /bsd.old
cp bsd /bsd

and reboot.

Then just read the main pages for raidctl to see how to set it up (hint: look at my raid5.conf above).

Hope this helps.

linux Software raid in general by russ_allegro · 2004-10-30 13:32 · Score: 1

I'ved use software raid 5 & 1 on over 6 servers with good success. I've ran several benchmarks on machines using software raid and hardware raid. They both offered compariable performance except in one area. If you constantly fsync such as unbuffered always sync syslog writes. Now this is slow no matter what but it was extremly slow in software raid.

This isn't really much of a problem as you don't really ever want to write to disk unbuffered always calling fsync.

I got me really bad when I turned up the logging in postgresql. While the performance degraded in both software and hardware raid. In the software raid it was really slow. However if you do buffered logging it is equaly fast on both hardware and software raid and almost no performance degradation.

So just remember don't do any unbuffered always fsync writes to disk in software raid. In most cases you'll never do this.

3Ware Hardware Works Every Time by SwedishChef · 2004-10-30 13:37 · Score: 1

We have built a dozen RAID5 file servers running Linux and used a 3Ware Escalade hardware controller every time. Every machine has worked flawlessly and continues to work. This includes our off-site backup system. Nothing else has worked as well and, for the $150 or so the card costs, it's well worth using.

--
No one ever had to evacuate a city because the solar panels broke!

PCI bottleneck by Mike+Hicks · 2004-10-30 13:37 · Score: 2, Interesting

I haven't read all of the comments in detail, but I think one thing that people are often forgetting is that a standard PCI bus has a theoretical maximum bandwidth of 133 MB/s, a level you'll probably never see in real life, especially when there's a fair amount of chatter on the bus from different devices (and you'd get a lot of that with 8 drives plus networking plus who knows what else). Of course, PCI bus layouts vary considerably between simple motherboards and high-end ones.

I don't know if anyone makes PCI-X ATA-133 controllers (non-RAID), so in the final analysis it might be best to get a 3ware card with a 64bit connector and plop it in a long slot. Of course, you need a pretty nice motherboard for that. I guess I haven't gone shopping recently, but they weren't that common the last I checked (and everyone is going to head for PCI-Express shortly anyway).

Of course, it all depends on what you'll use the machine for. If it's just file serving over a 100Mbit network, there's no need to worry that much about speed. It's only a big deal if you're concerned about doing things really fast. I believe good 3ware RAID cards can read data off a big array at 150-200 MB/s (maybe better). My local LUG put a ~1TB array together for an FTP mirror with 12 disks (using 120GB and 160GB drives, if I remember right) about 2 years ago, and testing produced read rates of about 120 MB/s on a regular PCI box (I think.. my memory is a bit flaky on that). Of course, I don't think anything was being done with the data (wasn't going out over the network interface, to my knowledge, just being read in by bonnie++ I suspect).

Re:PCI bottleneck by Anonymous Coward · 2004-10-30 14:57 · Score: 0

I have been wondering if a non-PCI (Intel CSA or nVidia MCP-S) on-board Gigabit would be enough to get good throughput without the expense of a server MB.

Software raid5 works well.. by jedimark · 2004-10-30 13:41 · Score: 1

I was running a P3 600, on an 810 board for a while running with 3 seagate 120gb drives for my brother who lives in a flat under my house. it was definitely not quick, but stayed up. The performance was pretty crap compared to the same hardware running a mirror, but a large part of that was the crappy controller on board.

I upped the box to an athlon 1200 on a sis mainboard and things improved quiet a bit. (both boxes had 256mb sdram) The raid and network performance doubled, and tons of delays disappeared.

After complaints (from the windows user) that it still wasn't utilising the bandwidth well enough, I reluctantly used one of my main PCs (athlon 2400, via kt266, 1Gb) and also tried an extremely cheap Sil680 controller (used just for the disk interface). The individual drive speeds went over 50mb/s. The raid5 was still only running at about 35mb/s. And of course no faster over the network - just a waste of a machine. :/

After not being able to stand the fan noises of the 2400, I split the drives off and put them in my brothers windows box as a mirror, seeing he's the only one who needed the reduntant space (he has some bizarre psychological need for 'warez' and mp3s) They failed within a day with loads of data recovery fun. He's since warranty swapped two of the for seagate sata's, and reinstalls once a month, and i've stopped caring about the plight of his O/S or warez.. :-)

These seagate drives were less than 5 months old, and were getting loads of seek & ecc errors from the start. The were all running with enough cooling (it's summer here 9 months of the year), and the temperatures were monitored constantly. Just crap I guess.

The moral of the story: use decent long lifetime drives from the start, of different batches, don't throw more money on hardware than you have to (a cheap faster ide controller does wonders), and use a decent O/S that is stable and lets you optimized to hell (ie gentoo :-)

Oh yeah, and kill all windows users.. ;-)

raid5 software is great by itzdandy · 2004-10-30 13:44 · Score: 2, Informative

in linux, raid5 is a very solid and fast solution. even with a 500Mhz, it is faster than all but the most expensive hardware cards, as most cards have a 133mhz chip or even less.

also, software raids are hardware independent. they can be modified easily while booted and without rebooting. if a hot-swapable drive is used, downtime can be eliminted by a hot-swap and a rebuild of a failed drive.

also, i have been in a discussion about the new cachefs patch in rescent mm kernel patches(or maybe nitro?), allowing you to use a cache in ram with any filesystem, so you could mount your raid array through the cachefs with a given amount of RAM for write cache :) should give a nice performance boost on many systems. this patch is designed to improve transfering files on networks but is show to work equally well for local devices.

AND, linux software raid works on a per-partition basis, so you can mix and match drive sizes without wasting space. 8 250GB drives can mate up with 4 300GB drives, and then the wasted 200GB can be made into another array.

you can easily add IDE cards and increase the size of your array.

you can spread your array over a large number of IDE cards for better redundancy, no single card will criple your array, and IDE cards are much cheaper than hardware raid cards.

LINUX can be booted from a software raid! while is has trouble on some hardware raids!(driver issues)

i run a software raid5 over 12 seagate 120GB drives with no problems. i get great transfer speeds accross the (gigabit)network and it's easy to manage drive spindown because the system sees each individual drive while hardware raid solutions typically only allow the system to see the array as a single device.

most hardware arrays are mainly configured at boot time. to build or repair an array, your system will not be working. if you run a linux fileserver/firewall, your firewall doesn't function on hardware raid rebuild, while it does in software.

--

though i would go with a faster processor, you should have very good luck, reliability and performance from an 8 device software raid5. and have a nice 1.7TB array

Spend the extra $200 and do it right... by patniemeyer · 2004-10-30 13:47 · Score: 2, Informative

I am very happy with my linux / 3Ware 4 port raid card combination. It makes it brain dead simple and takes linux out of the loop of things that could trash the raid. I even forgot to install the *drivers* for the raid in the initial install and it all just worked fine... because the box thinks it's one big magical drive. (The drivers were only necessary for monitoring...)

Spend the extra $200 on a 4 port card... put a *big* fan on the drives because that's the #1 killer and you'll be happy.

Pat

my experiences with software raid by Skuld-Chan · 2004-10-30 13:49 · Score: 2, Informative

It worked okay until one day I had to reboot my file server (moved locations) and I couldn't get the raid to come back up. I lost all my data :(. The bad part is - when it came to forums, irc and generally trying to get help there really wasn't any, a good amount of the documentation out there and the troubleshooting information is for the older tools. I generally believe that when it comes to your data you can only trust tools you can actually support - software raid for all intents and purposes seems highly alpha/beta.

Anyhow I bought a 3ware 7450 Raid controller and haven't looked back - its brutally fast (over 20-30 megs a second in a sequental write), fully supported in linux and it a piece of cake to setup.

Its not bad at recovering either - I had a power failure and the ups failed later on - machine restarted of course when the power came back on and the 3ware controller automatically rewrote all the parity on the disks - everything was fine. While it wrote the parity the system was up and running instantly (raid was in a fail state of course).

250gig hard drives on a 500mhz? by whistlingtony · 2004-10-30 13:53 · Score: 1

How are you going to connect all these hard drives? You mentioned software raid... If you're planning to use the IDE on the motherboard... A motherboard that old might not recognize that large a HD. Got Raid Cards?

Re:250gig hard drives on a 500mhz? by TheNetAvenger · 2004-10-31 02:17 · Score: 1

How are you going to connect all these hard drives? You mentioned software raid... If you're planning to use the IDE on the motherboard... A motherboard that old might not recognize that large a HD. Got Raid Cards?

Maybe this is just a little to weird, but, why is the mainboard age (BIOS/Recognition) of the Hard Drives even an issue?

At the very least the Mainboard (if way old, say like in the Pentium 166MHz chipset days), it would still see the drives as a 8gb Hard Drive at the very least.

Once the BIOS boots the OS, is not the OS responsible for 'seeing' the hard drives?

Heck, even the driver architecture of Windows NT(XP) has no regard to what the BIOS thinks the hard drives are.

As for the poster's original question...

Yes, Software RAID 5 works well, can be just as reliable as any Hardware RAID 5, and in most cases even have comparable performance, even in high end usage servers.

I can personally attest that even back in the old NT days, that we ran Software RAID 5 on 200 MHz Pentiums with 32mb and 64mb of RAM, with incredible performance and reliability.

Ignore the posters in other threads above telling you that need Mass amounts of RAM in a file server, this is only true if you are going to have tons of users hitting this server at the same time, if it is just a RAID 5 file server for the house or a small office environment, extra RAM isn't going to help much, especially if this server is going to be accessed via a network, the bandwidth is not going to out perform the speed of the Hard Drives or the RAID5, so the RAM, which would be used in a caching capacity would not be necessary or utilized.

Everyone here has an opinion, but the main thing to remember is, YES they all pretty much agree that Software RAID 5 is reliable, and do a little testing yourself, you will find any pitfalls that would be specific to your situation that others won't foresee.

Yeah right by Anonymous Coward · 2004-10-30 13:54 · Score: 0

Give us a reference to the standard used for on-disc storage of the RAID management information then.

The only "standard" here is what the RAID levels mean. Everyone implements the on-disc data storage differently.

For those looking by phorm · 2004-10-30 14:02 · Score: 3, Informative

smartctl often comes as part of the package "smartsuite." For Debian users there is an apt package available under that name as well.

Re:For those looking by Reez · 2004-10-30 21:41 · Score: 1

From the smartmontools homepage, it seems that smartsuite is its no longer maintained precursor, and that smartmontools is the way to go.

Even better smartd options by anti-NAT · 2004-10-30 14:16 · Score: 3, Informative

You can get smartd to execute tests automatically, using the -s option.

In my smartd.conf file, I have :

-s (L/../../7/03|S/../.././05)

on the device lines, which means do a weekly online long test at 3 am Sunday, and a daily online short test at 5 am every day.

mdadm running as a daemon, and watching the md arrays is also a good idea.

--
The Internet's nature is peer to peer - 20050301_cs_profs.pdf

Re:Even better smartd options by Anonymous Coward · 2004-10-30 19:57 · Score: 0

That option wins my "Most Perl-ish Option" Award for the day. Congratulations to yourself and the smartd developers!!
Re:Even better smartd options by Abcd1234 · 2004-10-31 05:46 · Score: 1

To you and the grandparent, a huge thanks! I wasn't aware of these tools, but now I have SMART monitoring enabled for the two drives which comprise my primary RAID. Thanks!

Fine by captaineo · 2004-10-30 14:29 · Score: 2, Informative

I have a 160GB Linux software RAID-5 consisting of three 80GB disks, running 24x7 for years now. (when I built the RAID, 80GB was the largest disk capacity you could buy :).

No problems at all. I once had an IDE controller fail - I replaced it (had to reboot of course), and Linux rebuilt the array automagically.

I have not tried using a hot spare.

Warning: a lot of the documentation out there on the web about Linux software RAID is very out of date. If you go this route, DEFINITELY buy the book "Managing RAID on Linux" (O'Reilly). Also be prepared to compile the "raidtools" package, which you need to set up arrays.

I have since added an 8-disk system based on 3Ware's 9000 series SATA RAID controller. I recommend 3Ware for higher-performance systems. (I have 8 250GB disks in a single 1.6TB RAID-5, I get about 180MB/sec read, 90MB/sec write.)

A few other hints by anti-NAT · 2004-10-30 14:34 · Score: 2, Informative

If you run smartmontools, you can configure smartd to not only monitor the SMART status of the disks, but also execute online tests - have a look at the "-s" option of smartd. For my RAID1 array, for each device, I have -s (L/../../7/03|S/../.././05) entries.

mdadm also has a daemon mode which can monitor the arrays, and if there are any failures, send an email to a designated email address.

--
The Internet's nature is peer to peer - 20050301_cs_profs.pdf

Do you know who AC is? by anti-NAT · 2004-10-30 14:38 · Score: 1

There is a pretty good chance he knows what he is talking about ...

--
The Internet's nature is peer to peer - 20050301_cs_profs.pdf

Re:Stick with hardware RAID, mod this up! by eclectus · 2004-10-30 14:53 · Score: 1

The advantage of HW raid over software is in the speed that HW offers you. Of course, if you are doing RAID 5, you usually aren't concerned with speed as much as redundancy.

--
This signature is a waste of 42 characters

raid5 + debian by POds · 2004-10-30 14:56 · Score: 2, Interesting

I'm running raid 5 on i think 2.6.8 with 3 drives. That is, i'm running it on the root partition and it runs alright, although, i have noticed it has goten slugish... maybe a defrag is in order?

When i started out, firefox was loading in 2 seconds and it now appears to be taking around 4 seconds to load. At least i think those mesurements are ok. If you want real speed, i'd think about using raid01 as it seems 4 discs in a raid0 array would be faster than 8 in a raid5? I'm not too sure about that, but raid5 is significantly slower than raid0 apparantly. Also, using those other 4 discs to mirror the raid0 array could be more usful then raid5s parity/crc redundancy.

--

Giving IE users a taste of their own medicine since 2005 - http://pods.-is-a-geek.net/

3ware 9500-8 by Anonymous Coward · 2004-10-30 15:04 · Score: 0

I just installed a 3ware 9500-8 w/ 8 sata drives running raid 0. I did it for speed. I'm backing it all up with an LTO-1 tape drive. I bought 3ware after ripping my hair out messing with some of the lower end hardware raid cards. It was hard to spend 525.00 on a raid card, but after I did, and got it working, I'm very happy with it. Using hdparm -t, I'm getting about 170 meg per second throughput. The 3ware site shows benchmarks of around 200 meg per second w/ western digital raptors (12 disks).. also, they use bonnie++ for benchmarking... just some thoughts..... I know you are looking mainly for file storage.... so your looking for a dependable raid 5... just my opinion, but speed isn't really an issue for you because you can only push about 8 megabytes per second down a 100 MB ethernet connection... go for the 3ware ... it's dependable, and they have open sourced thier drivers. They have the best linux support of anyone I've seen for raid cards. Good Luck !

RAID 5 and Redhat by lawpoop · 2004-10-30 15:11 · Score: 1

I setup a raid 5 on redhat about a year ago, and I was very concerned. After trying and failing to get it running with debian, I switched to redhat, which has the raid setup on it's graphical installer.

Then the nightmare happened. The computer (an old 400 MHZ celeron) stopped accepting keyboard input. No way to get past the boot error. I moved the raid array to another machine, keeping the drives in the same order ( drive 0 on the master on IDE 2, drive 1 and 2 on their own IDE channels on an IDE card). A an install of redhat saw the raid right away and set up the disk accordingly.

Apparently redhat marks each disk with a volume label, so it knew they were part of a raid. I don't know what would have happened if I got them out of order (who knows? Maybe I did), but I certainly feel a lot better about software raid under linux.

--
Computers are useless. They can only give you answers.
-- Pablo Picasso

Re:RAID 5 and Redhat by smcavoy · 2004-10-30 18:10 · Score: 1

just for your info, the new Debian Installer supports RAID and LVM on install and can even use them (lvm and raid1) for root.
Re:RAID 5 and Redhat by Anonymous Coward · 2004-10-31 01:21 · Score: 0

> Apparently redhat marks each disk with a volume label, so it knew they were part of a raid.

Maybe you mean the 'persistent superblock', which allows the kernel to read configuration of your RAID devices directly from the disks involved.
Very handy, and it's not a redhat exclusive.

Re:Compaq smartarray cards are backwards compatibl by Judg3 · 2004-10-30 15:15 · Score: 1

Indeed, they are.

In fact, I used to work in a datacenter with some 2500 Proliant servers running mostly the 6400 SA controllers. For about 3 years we had to shut down the datacenter completely about once or twice a year (Mainly for upgrades to the electrical systems) - and in bringing things backup we'd end up needing to replace 4-5 controllers - and they never skipped a beat! Same with upgrades, never had any problems.

--
Looking for hardware (Currently need: Large Etch-a-Sketch) Have one? See my journal!

RAID lowers sched downtime, NOT replace backups by BenRussoUSA · 2004-10-30 15:16 · Score: 1

Have fun with the RAID array, sounds like fun, but a little advice. RAID gives you much added protection against outages during critical hours. And it does save you from having to restore from backups *sometimes*

However most data loss occurs from HUMAN or SW errors, not HW. RAID won't protect you from either of those. If you want to have an online archive of music and movies and pic's etc... That is very nice, you don't want to loose all the time and effort you put into that.

I *assume* you have a broadband ISP connection. (cable, DSL?) You have a good friend who does too? If so, why not share the cost and the space with a friend, set up rsync between your houses for remote data backup on a regular basis. (There are howto's regarding this.)

Now you can lose a disk or a controller, or the whole server, and all you have to do to recover *everything* is set up a machine with at least as much disk space and setup the rsync to rebuild your copy from your friends house.

If you do this, you can save the worry over all the details and issues of RAID setup and failed disk replacement and such. (I mean do that stuff, but don't fret over it, because you will always have all the data backed up remotely anyway.)

RaidWeb.com has nice hardware too. by jbn-o · 2004-10-30 15:25 · Score: 2, Informative

I have used raidweb.com enclosures in the past and they work quite well. They handle all the RAID configuration inside the box and appear as one drive to the host (hence the boxes are totally host independent). The connection between the box and the host is SCSI and I've used off-the-shelf high-end SCSI controllers for this. Their boxes have redundant fans and power supplies. They sound like a jet taking off, but my experience is that they're stable and rock solid. They're rack mountable too.

The only big disadvantage I experienced at the time was the lack of docs on the serial controller, so I only had the audio buzzer signal to go on when a drive failed. I think the box would have sent a signal over the serial link to the host indicating a failure. Then the host could do something interesting with that signal like send e-mail, call a pager, and so on. It would have been nice to have remote signaling, but in this case I didn't need it. The install site always has someone there to handle taking out the bad drive and plugging in the cold spare.

--
Digital Citizen

Re:RaidWeb.com has nice hardware too. by illtud · 2004-10-30 23:52 · Score: 1

The only big disadvantage I experienced at the time was the lack of docs on the serial controller, so I only had the audio buzzer signal to go on when a drive failed.

Take a look at safte-monitor.
SAF-TE is an open spec from Intel. Quote:

SAF-TE: SCSI Accessed Fault-Tolerant Enclosures Interface Specification

This specification defines a set of SCSI commands for setting drive status information, including status for RAID arrays, into a disk drive array enclosure. The drive array enclosure may be a separate enclosure, or the same enclosure. The specification also defines commands for managing hot-swap drive slots and returning environmental health information for a drive enclosure.

Most (good!) SCSI RAID enclosures will speak SAF-TE, but not many people make use of it. Try it out with your own enclosures - you may be surprised!

RAID5 under linux 2.4 by Spazmania · 2004-10-30 15:29 · Score: 1

Under 2.4, software raid 5 is stable. Monitoring it is trivial (cat /proc/mdstat). Rebuilding the array after adding a drive is trivial (raidhotadd /dev/md0 /dev/hda1). Rebuild happens in the background on the fly and you can watch it in mdstat. You can even set the partition types to raid-autodetect instead of linux (type fd instead of 83 in fdisk) and have the kernel notice and build the raid5 device before trying to mount root.

I use 8x250gb Linux raid 5 at work for an online backup server. Its rock solid, a real champ.

Couple gotchas:

1) Only ONE parallel ATA drive per controller. NO SLAVE DRIVES. IDE drives can't disconnect the way SCSI drives can, so having parallel traffic to both a master and a slave drive will KILL your performance.

2) Beware of multiple Promise cards in older (P2 and P3) machines under Linux 2.4. Something has gone haywire in the driver. Works fine on old Pentiums and fine on Athlons and P4s but on all three P2 and P3 motherboards I tried (two different model intel boards and an ASUS P2B) I got rapidly increasing ERR counts in /proc/interrupts followed by eventual kernel oopses. A single drive channel worked flawlessly but simultaneous access to two drives = problems.

The CMD Technology Inc PCI0680 controllers (e.g. SIIG, Adaptec) worked great.

Remember: 8 drives, 1 drive per chain = 2 drives per card = 4 cards = 4 PCI slots. Make sure you have enough left for video and ethernet.

--
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.

gigabit fiber by bani · 2004-10-30 15:34 · Score: 2, Informative

why? gigabit copper is just as fast and ~50x cheaper.

Not really necessary for a desktop... by boola-boola · 2004-10-30 15:40 · Score: 1

Unless you're doing lots of long reads (big files, such as databases), it's not really necessary for a linux system. I believe the average file read in a linux system is what, under 1k? You're really only saving on sector transfers, but you're not gaining anything as far as drive head seeks or basic rotational latency.

I'm running a RAID0 array on 15k SCSI HDs here, and honestly it's _not_ that much faster, except for when I'm recording audio (those WAV files get big FAST). Then again, I could still do it with no RAID.

Overall, unless you're dealing with big files, you won't gain much. (not to mention if you have too many drives in your array and not enough throughput (e.g. only ATA100 with 5+ drives), you'll really only choke your system bus.

Re:Not really necessary for a desktop... by timerider · 2004-10-30 20:17 · Score: 1

try reading once in a while, it might even help you understand what people want...

the guy is aiming for a SERVER. that is NOT a desktop box as you thought. For the unwashed: a SERVER sits in the corner and dishes out files by the dozen.

linux software raid5 flaws by bani · 2004-10-30 15:41 · Score: 1

this is a fatal flaw in the linux software raid5 design.

real (read:industrial strength) raid5 designs dont mark the whole drive unusable, just the few blocks that had uncorrectable errors.

the linux raid5 driver really needs this feature, and i wouldnt recommend using it until it does.

Re:linux software raid5 flaws by Anonymous Coward · 2004-10-30 19:27 · Score: 0

The drive marks the bad sectors. The RAID controller has nothing to do with it -- it just moves the data to the new place.

Any raid controller -- software or hardware -- will mark a drive unusable when it becomes unusable. Losing a few sectors here and there is not going to make any drive unusable -- that's a normal part of the hard drive aging process and won't have any significant effect. The hard drive itself detects them, marks them, and moves on.

Don't recommend or diss something unless you have a clue.

Don't forget LSI. by Ayanami+Rei · 2004-10-30 15:43 · Score: 1

They make good RAID controllers too for a competetive price. (They OEM the Dell "PERC" cards)

--
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON

Re:PCI bottleneck and separate channels by BlueBiker · 2004-10-30 15:52 · Score: 1

Mod parent up, this is great advice. I've had superb performance and reliability from software RAID 0 and 5 under Linux 2.4 and 2.6 for the last two years using a fast/wide PCI bus.

Hardware is a trio of Seagate X15-36LP Cheetahs connected to a Tekram U160 card running in a 64bit/66MHz slot of a dual Athlon XP 2000+ Tyan Tiger MPX motherboard [currently $215].

When I moved this controller and drive setup to the standard 32bit/33MHz slot of a dual P3/450MHz machine, disk performance was drastically cut while CPU usage wasn't much affected.

Modern hard drives are pushing 70MB/s transfer rates, so even two of them can saturate the classic PCI bus. PATA, SATA, and SCSI can all support high RAID transfer rates if their controllers are connected to the system by wide-enough / fast-enough interfaces and smart choices are made about [not] sharing channels.

BTW, at least a few of the common Linux filesystems are smart enough to automatically configure themselves efficiently when formatted on top of a RAIDed partition.

Skimming the surface... by Ayanami+Rei · 2004-10-30 15:54 · Score: 1

It's better to have a combination of both hardware and software RAID. You can take better advantage of a larger server (especially with multiple I/O buses) if you add a couple of independant RAID controllers, then bind them together with software RAID. Raid 50 comes to mind... So you can treat each controller and it's respective drives as individual volumes in the metaraid, spreading out the chance of data loss due to an adapter dying; simultaneously getting a nice performance boost.

--
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON

Errr... by Ayanami+Rei · 2004-10-30 15:56 · Score: 1

RAID5 is almost always better when offloaded (unless you have like mad SCSI controllers for your direct-attached discs)

--
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON

Run software raid, but back up /etc/raidtab !! by subStance · 2004-10-30 16:05 · Score: 1

Completely agree, but I would add an important piece of advice:

If you use software raid, BACK UP /etc/raidtab !!

It seems like trivial info, but when I lost my os partition with a raid0 set attached, I had a hell of a time trying to figure out what the old raidtab looked like. If you get it wrong (eg in reverse order) when you try to guess, you can end up with directory listings that look fine, but you're corrupting the disk with every write.

Seems crazy, and it may have been an older incarnation of software raid, but I ended up having to re-rip 50 dvds as punishment. Make sure you back it up !!

--
Servlet v2.4 container in a single 161KB jar file ? Try Winstone

Re:Run software raid, but back up /etc/raidtab !! by Anonymous Coward · 2004-10-31 03:15 · Score: 0

This is actually a huge disadvantage. A good RAID system should be able to reconstruct the volumes automatically even if you shut the system off and moved around which controller all the disks are on. They do this using a controller independent drive ID and a configuration that's written onto a partition of each disk.

Buy a 3Ware card by rspivack · 2004-10-30 16:24 · Score: 1

You just spent $960 on the drives (8 x $120). So obviously, you can afford to spend a few dollars. So you probably can spend about $100 more and get a 3Ware raid card (afterall, you saved a lot on the drives themselves.) Since reliability is important, and you got a great deal on the drives (I hope you remember that only 1 mail-in rebate per address when you send in for your refunds), so spend a small fraction more to get something that works - hardware RAID with a real controller.

Raid w/external firewire drives? Does it work? by xtal · 2004-10-30 16:32 · Score: 1

I would be very interested in doing this (in the near future) but can someone confirm that this will work without any hassles in Linux? I tried doing a setup a long time ago with external drives and ran into all sorts of problems.

The issues were related to the external controller itself and not linux, but would like to find this out before I went down the road again.

--
..don't panic

Heat will be a problem by tylernt · 2004-10-30 16:41 · Score: 2, Informative

I've stuck 4 7,200rpm IDE drives in a case... and promptly killed two of them within days. I had to add a rear exhaust fan to the case, a PCI slot blower, and I removed the blanking panels in front of the drives to get more airflow. The drives now stay merely warm to the touch (instead of HOT), and the drives have been fine ever since.

8 of those suckers are going to get toasty without plenty of auxilliary cooling.

--
DRM 'manages access' in the same way that a prison 'manages freedom'

Re:Heat will be a problem by __david__ · 2004-10-30 17:58 · Score: 2, Interesting

Yes, heat is definitely an issue, and an issue I didn't even think about when setting up my 4 disk linux software raid 5 set.

After I set it up for the first time, I had a drive die on me really quickly and noticed when I replaced it that it was murderously hot. As in "burning my fingers" hot. So I went and bought these little hd cooling fans that fit in front of a 5 1/4" drive bay (and come with 3.5" drive mounting adapters) and have 3 little fans on them. They cost about $7 each. I put 4 of them in my machine and they kept the drives at room temperature. Ahhh.

But the noise was a problem as all those fans together sounded like a wind tunnel. Especially 2 years later when all the little fans started dying and making extremely loud noises. Think annoying fan noise multiplied by 12. Ugh. Then I found this neat product:

Cooler Master 4 in 3 device module.

Instead of 12 little fans I now have one big super-quiet fan, and my drives still say nice and cool. It was definitely worth the $30 I paid for it.

Don't forget about heat.

-David

--
There. Now go play some cool javascript games!
Re:Heat will be a problem by tylernt · 2004-10-31 01:59 · Score: 1

Wow, that's pretty ingenious! Thanks for the link. Only thing I don't like is the sleeve-bearing fan, which won't last long. I'm sure there are plenty of ball bearing 120mm fans you could replace it with, though.

--
DRM 'manages access' in the same way that a prison 'manages freedom'
Re:Heat will be a problem by jovlinger · 2004-11-11 05:05 · Score: 1

I just bought one, thanks to your tip. Precisely what I was looking for! (zipzoomfly was cheapest I could find, if you include shipping)

My best guess is that it would work fine. by Futurepower(R) · 2004-10-30 16:51 · Score: 1

Anger problem?

The question is, can two cheap power supplies be used in standard case? The question is NOT whether you can pay $2000 for special equipment.

Obviously both power supplies would be attached to the case with the standard screws. That would provide a hundreds amps worth of common ground connection. Obviously the black signal ground wires would need to be connected to each other with a small additional wire.

I am a real electronics design engineer. I can't think of any reason why this would not work. It would probably be necessary to shut off the power to the hard drives separately.

There would be a little common mode noise, however this is something that the signal ICs in the hard drives can handle.

--
24 wars since WW2: Creating fear so rich people can profit.

Re:My best guess is that it would work fine. by Anonymous Coward · 2004-10-30 17:17 · Score: 0

ok,

first: power consumption, you need to do a little math here, and possibly get some information from informed people, but a good quality 450W supply can support approximately 10 drives and the system....if you're going bigger than that you'll probably want to ask an expert.

second: what 2-bit college did you get an EE from? for christ sakes, "black signal ground wires" -- ? A) This is a DC circuit, look it up in a reference manual somewhere; B) If you don't understand A) then go back and get your money back from the school

third: I would *NEVER* suggest that any of the sheep who are hardcore cheap-assed freaks try to do a mod for doing 2 ATX supplies in a single case, don't do it, it's not that your life is worth anything, it's that you might cause a fire.

fourth (time for me to relay something 'nice'): If you want dual supplies and you're too cheap to spend about $450-500 for a case and psus designed for the job then at least look to an AT PSU as an alternative, remote mount the power switch using a case mod in the front of the box (power up the drives before the system) and you're set, use a chassis ground for each psu, along with a good earth ground and you will be set. And, no there won't be noise problems, but I can't suggest doing this when you can step into an engineered solution for $2000...

See aren't I nice...You have your advice, but remember, only you can prevent nerd fires

Not so much by Anonymous Coward · 2004-10-30 17:11 · Score: 0

I don't know how's it actually done, but it seems to be the parity generation could be done by a on-the-fly counter that tracks bits as their written. It could be done really fast in hardware too. Not so slow in software either. Of course, I'm just throwing stuff into the discussion coz I'm a little curious, and don't know much about the issues in implementing RAID and large arrays. ... Enlighten me?

Optimal Configuration For Linux Software RAID by yanfali · 2004-10-30 17:28 · Score: 1

I used to worry about how to recover. I've had to do recovery a number of times, you can use a LiveCD, you can get an external USB HD. But the best configuration I've found so far is to get a Compact Flash to IDE converter, install it as /dev/hda and put /boot and grub on the CF card.

Then I install the OS on a RAID1 set and the data I care about on a RAID5. One system I have has dual SCSI 2xRAID1 for the system partitions and 3xRAID5 IDE drives.

The CF is /dev/hda and is mounted read-only. If I need to upgrade the kernel I remount it RW and then do what I need and remount back to RO.

This is the most reliable configuration I've come up with so far. CF has no moving parts so it's fairly reliable. Writes are limited to about 1 million. Reads are unlimited.

Compact Flash Readers
http://store.yahoo.com/ituner/cfdisk1c.html
http://store.yahoo.com/ituner/cfdisk5d.html

Some boards don't like the 5D type Converter, since they detect using the cable type (40 or 80). In which case type 1C works just fine.

This guy has already done it... by Anonymous Coward · 2004-10-30 17:35 · Score: 0

...and found hardware to be a good thing.

http://www.finnie.org/terabyte/

Re:CONFIGURE IT RIGHT!! small parts... by k12linux · 2004-10-30 17:39 · Score: 2, Informative

Actually, I can see some sense to that. He did mention failing during rebuild. That's when we are at the greatest risk of another failure after all since they are working harder than normal.

If you have one large partition and impending drive failure wipes out any cylindar on that drive, all the data on it is shot. That drive won't be used at all during the rebuild... a rebuild of 250Gb. You are at risk if, during any time of the long rebuild, a 2nd drive fails completely or even coughs up a bad cylindar which can't be redirected.

If you have 6 partitions, only the "damaged" one has to be recovered immediately. Obviously you would want to recover them all as soon as possible since that first drive is probably going to bite the dust soon. Even if you do lose the first drive completely and then a 2nd drive during a rebuild, you at least may not lose everything. Any of the 40 Gb blocks which were rebuilt before the 2nd drive died would have been saved.

Getting a slightly different sized drive for an RMA can also be a problem. What if your original 250Gb drives were actually 250.3Gb and the replacement is 250 Gb even? You aren't going to fit that single 250.3 Gb partition onto the replacment drive. And are you going to call the drive manufacturer and complain that your original drives were too big?

I've had issues with this on hardware RAID. I had to back up 600Gb over the network, wipe the entire array out, rebuild it and restore the data. If it had been software RAID, I could have backed up the data from the last partition into one of the others just to be safe, resized the last one, reformatted and copied the data back.

LVM with multiple partitions would have made it even easier.

Our backup servers are all software raid5 by detain · 2004-10-30 18:04 · Score: 1

We have 4 backup servers currently all either 120gb x 8 or 200gb x 8, on anywhere from 600mhz-800mhz processors w/ 256mb ram each.. they are under heavy i/o 24/7 and have performed great. hdparm tweaks are definitly helpful. Throughout our various drive problems in the past software raid has been great to recover from. We've even managed to recover from loosing 3 drives already at once in a raid5 array w/ 1 parity and 0 spare. The systems often have a fairly high load so they are only useful as a fileserver-only type setup but work great. software raid runs incredibly stable and we've been very happy with it

--
http://interserver.net/

Linux raid-5 is solid by __david__ · 2004-10-30 18:09 · Score: 1

I've been running it at work for 3 or 4 years now and at home for 2. At work we have a 3 disk raid array and at home I have a 4 disk raid array. I've had multiple crashes, disks that went out and power-failures and haven't lost either raid set yet. I've had some close calls, but I was always able to salvage the array, fsck it and move on.

The one really cool feature that's been added to linux recently is the ability to grow your raid. My original drives at home were 120 GB each. I have been slowly buying replacement 200 GB drives for my raid set over the past few months (as my budget allows). Each time I'd buy a new drive I'd mark it as failed in the raid, shutdown linux and replace the drive, and then power back on and hot-add the new drive. After all 4 were replaced I ran the new version of mdadm with linux 2.6.8 and grew the raid set:

[david@death ~]$ sudo mdadm --grow -z 195357377 /dev/md0 [david@death ~]$ cat /proc/mdstat Personalities : [raid5] md0 : active raid5 sdb1[0] hdg1[3] hde1[2] sda1[1] 586071936 blocks level 5, 128k chunk, algorithm 2 [4/4] [UUUU] [============>........] resync = 60.0% (117284488/195357312) finish=58.7min speed=22104K/sec unused devices: <none>

Suddenly my raid set had grown to cover the full 200 GB of each drive. I just have to say, that is really cool!

-David

--
There. Now go play some cool javascript games!

Don't forget backups by javaguy · 2004-10-30 18:16 · Score: 1

What if the house burns down, you're robbed, or a tornado destroys your house? Bye bye to all your data. Don't forget you need to back up your most important data, if not the whole array.

Hardware raid IS software raid in disguise by Anonymous Coward · 2004-10-30 18:26 · Score: 0

Hardware raid is nothing but software raid with a cheaper/weaker CPU (perhaps ARM or MIPS), and software (firmware) that is hard to patch.

Dedicating a 500MHz pentium to be a file server will outperform juat about all HW raid systems.

The only excuse I'd see for HW raid is if your machine is busy doing other stuff besides serving files (like if your fileserver _is_ your renderfarm) - but if that's your approach you probably have bigger problems anyway.

Re:Ok. by techsoldaten · 2004-10-30 18:38 · Score: 1

'maxtor rules': 118,000 hits

Not that Google searches are an accurate gauge of anything.

M

This is a very flawed logic by rpwoodbu · 2004-10-30 18:46 · Score: 4, Informative

This logic doesn't hold. Let's first talk about the performance.

Also, on any reasonably modern system, the software RAID will be faster. You just have a much faster processor to do the RAID processing for you. The added overhead of the RAID5 processing is nothing compared to a 1-2GHz processor.

The actual RAID processing is relatively easy, and any RAID solution, be it hardware or software, that is worth anything will not have any trouble doing the logic (perhaps the cards mentioned are indeed not worth anything). The processing isn't your limiting factor; it is data thoughput. This is where hardware shines. A lot of extra data has to be shipping in and out to maintain and validate the RAID. This can easily saturate busses. A hardware solution allows the computer to communicate only the "real" data between itself and the hardware device, and then allows that device to take the burden of communicating with the individual drives on their own dedicated busses. Sure, that device can become overwhelemed, but I submit to you that if it does, it was poorly designed.

I am not saying that one shouldn't consider software RAID solutions. Just don't consider them because you think the performance will be better.

Now lets talk about data recovery.

I've lost 4 drives out of a 12 drive system at the same time, and Linux has let me piece the RAID back together and I've lost nothing. Was the machine down? Yes. Did I lose data? No. Compare that with a 3ware hardware RAID system where I lost 2 drives. Even thought I probably could have salvaged 99% of the data off that array, the 3ware just would not let me work with that failed array.

Let us be clear: we are talking about RAID5. In RAID5, you simply cannot lose more than one drive without losing data integrity. And it isn't like you can get back some of your files; the destruction will be evenly distributed over your entire logical volume(s) as a function of the striping methodology. So it is quite impractical to recover from this scenario. I don't know what kind of system was being employeed with this 12-drive array that can withstand a 1/3 array loss, but it certainly wasn't a straight RAID5. I can come up with some solutions that would allow such massive failure, but then we aren't comparing apples to apples. I'd be very interested in knowing what the solution was in this example case. It should also be noted that we don't know how many drives were in the system that lost 2 drives, much less what kind of RAID configuration was being used. No conclusion can be derived from the information provuded.

As an aside, more often than not, when we as individuals want a large cheap array, we are less concerned about performance than reliability. We put what we can into the drives, and we hope to maximize our data/$ investment while minimizing our chances for disaster. A software RAID5 is a good solution. Some posts have said that if you can spend so much on the drives, what's stopping you from spending on a nice hardware controller? I submit that perhaps he's broke now! And besides, a controller that can RAID5 8 drives is quite the expensive controller indeed. This has software RAID written all over it.

Re:This is a very flawed logic by Anonymous Coward · 2004-10-31 00:11 · Score: 0

If the 4 drives fail in different places, or on some unused space, will the h/w raid be able to recover? Obviously, with s/w raid, it might quite possible to recover data.
Re:This is a very flawed logic by rpwoodbu · 2004-10-31 06:49 · Score: 1

No, there will be no difference between the recoverability of an equivalent software or hardware RAID. Now, some solutions might offer more options in the recovery department, such as trying harder to get data off of a failing drive before giving up, but that is not a function of whether the solution is based in software or hardware.
Read up on how the various RAID solutions work. In short, RAID5 is an "n-1" solution, meaning that you get the total storage capacity of n-1 drives, where n is the number of drives in the array. So in a simplified view, one drive is used as the parity drive (e.g. redundant data), so only one drive can be lost without invalidating the array. If the implementation is RAID5, this is the way it will work, regardless of whether it is done in hardware or software.
Bear in mind that you can do things like hot spares and mirroring that can give you an extra level of protection in addition to RAID5. But it does mean commiting more drives without getting more data capacity.
Re:This is a very flawed logic by the_hose · 2004-11-07 07:21 · Score: 1

"This can easily saturate busses. A hardware solution allows the computer to communicate only the "real" data between itself and the hardware device."

Well, that really depends. For sufficiently large io, that should end up being only a small amount of additional data for parity.

The fundamental problem with hardware raid ends up being the fact that your bandwidth to disk is bottlenecked at the raid controller (or host bus iface, if you're using an external raid box). If you have the opportunity to spread your raid out over multiple busses (say, a simple add-on PCI-ATA controller, an onboard ATA controller, an onboard RAID controller), that may be a better option. In most modern chipsets, these disk controllers are on independent busses off of the much higher bandwidth interconnect to cpu/memory.
Re:This is a very flawed logic by rpwoodbu · 2004-11-07 08:03 · Score: 1

Yes, I agree that it all "depends". If you have plenty of spare bus time, then it won't be a major issue. And indeed, there are things you can do to spread it all out. And yes, sometimes using a controller card instead of the built-in chipset controller can move data that was on a dedicated internal PCI bus onto the more crowded PCI bus for external devices. There are no hard and fast rules in this; good judgement must prevail.
However, I must disagree with your assertion that parity will be a small amount of additional data. Again, it all depends. For a RAID5 using 3 drives, the parity consumes an extra 50% of data (e.g. if three drives gives you two drive's worth of storage, the other third must be parity, hence 50% more). While "small amount" is a subjective measure, 50% seems a bit steep to consider small. If you increase the number of drives in your array, you decrease the proportion of parity data in the array, which would eventually reach a "small amount" by anyone's standard. In reality, most RAID5 implementations use three drives. In the case of this Slashdot story, he is using many more drives, thus the partiy data would indeed be a small amount. Therefore, a combination of his chipset controllers and add-on ATA contoller(s) on seperate busses could spread the load quite effectively in the case of an array with many drives.

Good, cheap controller? by Anonymous Coward · 2004-10-30 18:55 · Score: 0

but where can you find a good, cheap, controller? via and intel southbridges are great, but promise controllers are really, really, flakey.

RTFM and search Google by Anonymous Coward · 2004-10-30 19:23 · Score: 0

RTFM and search Google before you start bragging about your 8 250GB disks you bought.

"Hey, I just bought a $5000 B&O sound system with $6000 speakers. What do you think is best: placing my speakers to the left, the middle or more to the right. Would they produce better sound when silver-plated, or can I leave them gold-plated as they are now."

Well, if *BSD wasn't dying, you could... by B747SP · 2004-10-30 19:29 · Score: 1

... implement the RAID-5 part of the step-by-step howto in this article. It worked great for me.

--
I find your ideas intriguing and I wish to subscribe to your newsletter.

The "signal ground" is important... by Futurepower(R) · 2004-10-30 19:31 · Score: 1

Your behavior is offensive.

What is worse is that you are pretending to know something about the subject, and thereby retarding a true discussion.

The "signal ground" is important because there are wires from the power supply that allow the motherboard to control the power supply. That's the "signal". Probably it would be best not to connect those lines to the motherboard, but to install a separate switch for the hard drives.

I'm hoping someone with some experience with this will discuss it.

Re:The "signal ground" is important... by Anonymous Coward · 2004-10-31 00:02 · Score: 0

first, get rid of your sensitivity, it has no place on a public board, if you can't take it, go away, yes, I am a great big troll and very proud of it, if you're offended, I must have struck a nerve and that relays that part or all of what I have said about your lack of knowledge is true...

second, *I* do have experience, something that you obviously have none of with the components especially, but electrical engineering also.

Now, you need some critical reading skills... DO NOT connect the AT PSU to the motherboard, it is unnecessary and A) would require an ATX PSU (reference previous post - a mistake for anyone except those who actually have the knowledge of DC circuits to implement) B) would be a waste.

You're referring to a "signal ground" -- are you talking about the data signal? If so, reliase that you're talking about 2 different closed circuits. There is a power circuit - a closed circuit within the AT PSU system; there is a signal circuit - a closed circuit wherein the power is drawn from the motherboard (and therefore from the ATX PSU system), they are -electrically isolated- from the power. These are different systems, not the same, there is not a common chassis ground between them...

wow. yes, I am offensive, yes, you are my latest target, but I have relayed advice - and if you haven't picked up on it, then it your loss.

Also, SW RAID is partition based by Admiral+Burrito · 2004-10-30 20:02 · Score: 2, Insightful

Another advantage is that software RAID allows you to use any kind of disk as a RAID element.

Also, it's partition based, not disk based (under Linux, at least). This means that with just two drives you can create one two-disk RAID-1 array (for safety) and one two-disk RAID-0 array (for performance). Just create two partitions on each drive, pair the first partition on each drive in a RAID-0 config and the second partitions as RAID-1.

You can't do a single RAID-1/0 array with only two disks though. You could try, but you wouldn't gain anything (in fact, you'd lose).

Don't be tempted to go hardware RAID. by rew · 2004-10-30 20:05 · Score: 1

Hardare raid has reasonable performance, works as it should in simple cases, but when the shit really hits the fan, you're totally screwed.

As a hardware RAID array doesn't have a console, and no option to "ask questions" to the user, it has to automatically DO things as drives are replaced.

With Linux software raid, you end up with the option of TELLING linux what the config is, and possibly recovering some part of your data, should things go seriously wrong. In a hardware RAID, you're totally dependent on the firmware in the card to "do the right thing".

Suppose you configure 7 drives as a RAID, and the 8th as a hot spare. The firmware will mark the drives with a config version and drive number:

1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7 and hot spare 1.0

Next drive 3 breaks.

1.1, 1.2, XXX, 1.4, 1.5, 1.6, 1.7 and hot spare
1.0

and the stuff is reconfigured:

2.1, 2.2, XXX, 2.4, 2.5, 2.6, 2.7 and 2.3

If now the person handling the drives makes a mistake and removes the 4th drive: 2.4 instead of the the third:

3x.1, 3x.2, XXX, 0.0, 3x.5, 3x.6, 3x.7 and 3x.3

so now whe have a broken raidconfig: "3x". No amount of replacing 2.4 will make it fit in, as the array has determined that the raid has been broken.

If you get this far, "datarecvery" is the only thing that can help. Don't write a byte on any drive and try to get things right again. Drives that can be jumpered to "readonly" would be a help. We comercially would just make an image of all the drives and work from there....

Stick with backups, single box == single point by xixax · 2004-10-30 20:34 · Score: 1

I don't care how good your RAID is, or if it's hardware or software. I have seen lightning fry every single component in a box. RAID makes it easy to recover from individual disk failures (which probaably account for most, but not all disk failures), but I'd still be doing some form of backup. Put some of the disks elsewhere and mirror your data to completely different site (if possible).

From that perspsective (that RAID lets you treaat most hardware failures as a nuisance rather than a catastrophe), hardware RAID might be attractive.

Xix.

--
"Everything is adjustable, provided you have the right tools"

I bought one of these from Newegg. by Futurepower(R) · 2004-10-30 20:40 · Score: 2, Informative

I bought one of these from Newegg. I had a lot of problems with it. I called Silicon Image technical support. They told me that particular chipset did not work correctly, and they would not release working firmware for it.

I told Newegg about this, but they continue to sell them.

Fry's sells them also. I told a Fry's manager that Silicon Image told me they know they don't work correctly. Fry's still sells them.

I would love to find a technically knowledgeable and honest distributor.

My experience by Mike+Markley · 2004-10-30 20:41 · Score: 2, Informative

I've got a 4x160GB SATA software RAID-5 array (about 450Gb usable) serving up files on my home network right now, running under the 2.6 kernel.

These drives are all crammed into an old Dell that was my Wintendo a couple of years ago. A few months back, the grilles on the drive-bay coolers I installed got clogged up and I lost one of the drives to overheating. Upon replacing the drive, the rebuild took the better part of an evening (but didn't need to be attended). No lost or corrupt data.

The only major problem I had was that the RAID was dirty in addition to being degraded (insert "your mom" joke here), because I brought my machine down hard before realizing what was going on. In theory, I could have done a raidhotremove on the bogus drive and brought things down normally

I ended up having to do some twiddling to get it to rebuild the dirty+degraded array. I don't remember what that was, but as long as you don't do something boneheaded like ignore kern.log messages about write errors to a specific drive, get annoyed that it's taking so long to cleanly unmount the filesystem, and hard-reset the box, that shouldn't be an issue :).

Re:Stick with hardware RAID, mod this up! by Anonymous Coward · 2004-10-30 20:54 · Score: 0

Speed used to be an issue. It isn't any longer. Modern CPUs can do the XOR calculation faster than most hardware based RAID controllers. Hardware's advantage is that it's easier to configure, you can boot off the array without any problems, and the OS doesn't need to be aware of the RAID.

RAID-5 data recovery after losing 2 drives by Admiral+Burrito · 2004-10-30 20:59 · Score: 4, Informative

No, you're not "done period". You'll lose a lot of data, but may still be able to recover some. Likewise when losing one disk in a RAID-0 setup.

Any file that resides entirely outside of the gap in the array can be recovered. How likely that is depends on the details of the filesystem, the striping, and the size of the file (the larger the file, the more likely that a part of it fell into the bit bucket).

Also, not all drive failures are total. You may have a RAID-5 array with one drive that completely failed, and another drive that just has some bad sectors. In that case you should be able to recover most of your data. Or you may have two disks with just a few bad sectors, which is even less bad.

This all depends on being able to force the array to allow access to the device, so that you can mount the filesystem (in read-only mode) and sift through the remains. Some (many? most?) RAID implementations may just give up if two disks in a RAID-5 array (or one disk in a RAID-0 array) are flagged as bad, in which case you really are screwed, even though your data is still there. From what people have been posting here I would guess that Linux SW RAID will let you force it, though I've never needed to try it myself.

Re:RAID-5 data recovery after losing 2 drives by pjrc · 2004-10-31 03:37 · Score: 1

This all depends on being able to force the array to allow access to the device, so that you can mount the filesystem (in read-only mode) and sift through the remains.
3ware's 3DM utility allows this with the older 7000 series boards. Presumably it also does with their newer ones, since this capability has been present for many years.
On the screen where you instruct it to rebuild the array, there is a button that forces it to ignore errors and rebuild anyway.
I personally experienced this with a card that went bad and was getting lots of checksum errors reading and writing. 3ware's tech support folks replaced the 3 year old card without any hassle.
I can also tell you from personal experience that once the arry goes into degraded mode, you can still access it even though the controller is experiencing errors reading from the remaining drives. Sometimes it will retry many times and eventually you get an I/O error reported by the OS. But 3ware does NOT suddenly give up when the array is degraded and one or more of the remaining drives aren't reading properly. You get the best effort it can make, but when it can't read a file because of errors, you get an OS-level I/O error, not sudden and complete inability to access all data.
Everything I have written here IS from personal experience, with the older 7000 series cards. I have not used the newer 8000 series (which appears to be the 7000 with SATA), nor the newest 9000 series, but there is no reason to suspect they have gone from a very sane and workable system to something so brittle that it suddenly stops allowing any access to any drive upon the first sector error when the array is degraded.

--
PJRC: Electronic Projects, 8051 Microcontroller Tools
Re:RAID-5 data recovery after losing 2 drives by flsquirrel · 2004-10-31 08:02 · Score: 1

The issue here was that hardware raid is no less recoverable than software raid. Bad sectors tend to creap up a little at a time. If you have a drive showing bad sectors (that is the drive has run out of spare sectors to remap) you damn well should replace that drive ASAP. There is no real excuse for the partial failure of multiple drives that you discribe other than incompetence on the part of the admin. It doesn't matter if it's hardware or software raid.

As for total failure on two drives, it's conceivable that some files would not span across more than one failed disk. However, then there's no way to check the integrity of that file either. If you're looking at total failure on more than one disk, more often than not it's either a heat or power problem which likely means that other drives are suffering or not far behind and likewise cannot be trusted. This iswhere we start talking about things like raid 50 across seperate enclusres (in case of heat problems from ventilation failure and certainly disperate powersupplies), but the issue here is SW raid which is quite impracticle for any reasonable sized raid 50 array in software.

Likewise, files that small probably should not be on a raid 5 array anyway since they're going to cause all sorts of "small write" problems. Something that noone seems to understand is that raid 5 is not an all perfect solution. If you do not write a file the whole way across the array, that is hit every disk except parity, you have to read the other cluster that you didn't write to in order to calculate the new parity.

Of course, small writes are a significantly smaller problem on hardware raid than software since much of the redundancy from the reads and checksumming invovled with small writes is contained locally to the card and does no require the intervention of the cpu thereby keeping quite a bit of unecessary data off the system bus.

If you have quite a few small files and more than about 3 maybe 4 drives modern fast drives tops in your raid 5 array, you're going to kill your array performance on SW raid 5. Perhaps you should look at raid 10. as a more small file friendly solution as there are less disks to read and no checksumming to perform.

As for "forcing" the sytem to mount a damaged disk, total drive failure implies that the disk is well....totally failed. Let's stop and think carefully about this. That means it can't be read. I am not aware of any raid controllers that won't continue operating with a disk that has bad sectors in order for you to continue revovering what you can (virtually the whole array). In the case of total failure, most cards have some form of maintenance or forced mode in which they will continue to operate in an unrecoverable state so that you can salvage what little is there.

Now, lets discuss raid 0 and recovering files. Let's assume a striped set of just two disks since you'd be insane to stripe more than that. Too many points of failure, data integrity and all those problems. I just checked. A completely blank MS Word file (I know, I'm a heretic for not using Open Office) and that comes to 23.5 and an even 24k on disk. Now, golly, wonder what the cluster size is on my disk. (Counts on fingers) I reckon it must be 8K. Now I have a realatively small 60gig disk. Lets say that 16k clusters are more on the average for most people now. That means that even for a totally blank Word document, you need two full clusters. Hopefully I don't need to explain that the way striping (raid 0) works is that every other cluster goes on the other disk. Just how many many modern document types do you think won't span the striped set. Certainly no office documents and there aren't many source files either.

Raid 5 makes things a little more complicated since the most basic raid 5 set would be similar in situation with 2 data disks and 1 parity disk. One thing to point out, as the number of disks in a raid 5 set go up, so does the chance to recover small files. However, so does the overall volatility of the set as there a
Re:RAID-5 data recovery after losing 2 drives by flsquirrel · 2004-10-31 08:19 · Score: 1

Before someone jumps on the chance to correct me, I mean raid 15 not 50. It's a sunday, I think I'm going back to bed now ;-)

Re:CONFIGURE IT RIGHT!! small parts... by cowbutt · 2004-10-30 21:19 · Score: 2, Informative

Hard drives have spare sectors set aside for sectors that die, and they are automatically remapped. If software RAID is detecting errors, just REPLACE THE DRIVE. The entire drive will die soon anyways.

Not quite. In my experience, bad sectors are only remapped by the drive firmware on write. Attempts to read bad sectors will return errors. This makes sense if you think about it; you might be trying to recover data, and the sector might be readable once in a hundred tries, but if you're writing to the sector, then obviously, you don't care about the data that's there already, so it's an opportune time to remap it.

--

Question for software RAID people by julesh · 2004-10-30 21:31 · Score: 1

I recently attempted to set up a software RAID mirroring scheme on my Linux box (late 2.4 kernel; I forget which and I don't have access at the moment). Because I already had data on one of the disks I wanted to use, this is the technique I used to set it up:

- partition new, empty disk to create a partition the size I wanted my mirrored array to be
- set up /dev/md0 as an array with the newly created partition and a 'missing' device
- copy the data onto /dev/md0
- repartition the old disk
- add the partition on the old disk to the array

The system then did a rebuild, copying the data from the new partition to the old one as desired. It worked perfectly until I rebooted, at which point it forgot about the second disk.

Any suggestions what I might have done wrong?

Advice: Don't do it. by ckaminski · 2004-10-30 21:47 · Score: 1

Save yourself the trouble and don't do it.

Build yourself a bunch of mirror sets, and merge the volumes using LVM/LVM2, and a journalling filesystem.

Trust me, RAID5 corruption SUCKS FUCKING ASS.

At least with RAID 1 mirrors, I can pull half the drives and continue working.

Unreliable by vlasky · 2004-10-30 22:09 · Score: 1

Back in 2003, I ran a Software RAID5 setup under Redhat 8.0 which I later scrapped.

The server had four Seagate Barracuda IV 80Gb drives (RAID optimised versions) connected to a Highpoint Rocketraid 404 controller. The system was a dual Athlon SMP configuration on an ASUS A7M266-D SMP motherboard. The system had 512MB of registered DDR ECC memory

When extracting large tarballs, I would often experience CRC errors at random points within the archive.

When I compared the copy of the archive on the RAID5 filesystem with an original copy, I found differences. This really freaked me out as it may have meant that errors were being randomly introduced into newly written files.

The problem disappeared when I manually disconnected one of the drives, placing the array in degraded mode, and then recopied the original archive.

This led me to suspect that the authors of the RAID5 code in the kernel did not get the implementation of the read/reconstruction algorithm quite right or that the code had nolt been tested properly under SMP configurations. I did not have time (or the patience) to investigate this further.

I abandoned RAID5 and switched to a nested software RAID-1+0 configuration. It works, but I had a bad experience reconstructing the array when one the drives gradually started failing.

Fortunately, my workplace has set aside some money to replace this machine with a new Dell server with inbuilt hardware RAID5 and hot-swappable drive enclosures.

My final advice is: if you have the money, don't settle for second best.

Booting with Software RAID 5 by MrHim · 2004-10-30 22:09 · Score: 1

One tricky bit is booting. LILO can't find your kernel if you use Software RAID 5, which leaves you in a bit of a chicken and egg scenario (you want to put your kernel on a RAID 5 partition for the same reason as everything else, presumably). What I did was buy a 16MB compact flash card and a $15 CF TO IDE adapter from Here. I then put the kernel on the CF card and used syslinux as the boot loader. Works like a charm.

Also, as others have said, you may be SOL if you're using hardware raid, the controller dies 3 years down the line, and the manufacturer isn't willing or is unable to get you a replacement. eBay can be a life saver in this situation, but having gone through that I don't care to do it again.

I'm using Software RAID 5 with 5 SCSI disks and I've been VERY happy with the performance and reliability.

Errr... by phrasebook · 2004-10-30 22:25 · Score: 1

Try reading other posts in this discussion.

Important note by phek · 2004-10-30 22:26 · Score: 1

I've scanned through about half the main posts here and haven't seen the following mentioned. Assuming you're talking about a software raid 5, you can't boot it. You need to set up your root partition on a mirrored software raid or no raid. This includes, /, /boot, /etc, /bin, and /sbin. As far as hardware raid goes, dont. Hardware IDE RAID controller drivers are notorious for their instability, so I'd strongly suggest the software raid.

So basically the setup you're going to want is to create at least 2 partitions per disk, one for /, and then another for /home, /usr, /var, and /tmp. It would be even better though if you created a seperate partition for each of those. When I'm dealing with an excess amount of disk space, I generally create a 2 gig root partition (though 1 gig should be enough). Now you're probably going to want to put this mirror'd setup on 3 of the disks (1 as a hot-spare), then that partition space across the other 5 disks for whatever else, i'll just say swap, though 10 gigs of swap is completly rediculous. Also a note on swap space, swap is an automatic RAID, so if you create 2 50 meg swap partitions, the kernel will combine them and access them similar to a raid0.

Here's another important note. Those cheap drive's from Fry's seem to have a high failure rate. I bought 4 of the 160 Gig maxtor boxed drives from $59 each about 9 months ago from there, and within a month 1 drive failed. A friend of mine had bought a few of them too and he also had one fail within a short time. The other 3 drives for me haven't given me any problems since then though. I had set mine up on my existing workstation as a RAID 5 with an XFS filesystem on dm-crypted partitions which turned out to be a horrible combination. Whenever I had a power failure I would lose all the data due to the way XFS stored its MBR. Switching to ext3 solved this problem. Anyways, even after the disk failure though, the data was fine (didn't even notice the disk failure until about 2 months later). Oh yah, the reason I mentioned this is because if you're getting 8 cheap disks from Fry's, there'll be a good chance 2 or more of them will fail within a short period of time, so I would definitly think about using one or two of them as a hot-spare.

Anyways, for details on how to set up a software raid, check out the linux software raid howto (google it), and it'll give you step by step instructions.

controllers by Anonymous Coward · 2004-10-30 23:43 · Score: 0

I've been doing sw raid5 on maxtor max attach machines running linux. It runs great. The only thing is that you MUST follow rules of IDE software raid, 1 disk per device. Your going to need at least 4 IDE controllers if you intend to put all 8 disk in the same puter. All masters, period. Do not bother with slave disks in the array. And save yourself some trouble and use a distro that can install on RAID.

RAID still requires backups! by Anonymous Coward · 2004-10-31 00:21 · Score: 0

The first rule of RAID is: Do not forget you still need backups.
The second rule of RAID is: Do not forget you still need backups.

Your RAID array will not help if your machine is stolen, struck by lightning, multiple-disks fail simultaneously, and so on.

Having spent around $1000 on disks, you might want to consider putting a similar setup in another friend's house, and doing automated backups of important files, so when the unthinkable happens and your RAID array fails, you still have your most important data.

6 drive software raid 5 with encryption by swilver · 2004-10-31 00:23 · Score: 1

I have the above setup running on my home linux system, a 2.4 GHz P4. The performance is around 10-15 MB/sec reading and writing. The CPU is the bottleneck because of the 256 bit blowfish encryption I use. It is however not relevant as the server is only equipped with a 100 mbit network card.

The most important thing when setting up such a server is to have sufficient cooling. I have 7 drives mounted in this server, and apart from the PSU fan I installed one extra fan. I used a midi tower case, with 4 drives installed in the top 3 5,25" slots, and 3 more (with atleast 0.5" space between them) in the 3.5" bays below. I left the top 5.25" slot and the floppy drive slot open for better ventilation (I removed the CD-ROM and floppy drive as they're not needed).

Cooling is very important, without keeping some of the slots open, temperature of the top drives could easily go above 55C, which is the max operating temp of the drives I use. Operating them above this will significantly reduce their life span. With the slots open I can keep them all below 40C.

Temperature is easily monitored with SmartMonTools. This package can also be used to run regular self tests of your drives, which will help detecting drives that are about to fail (happened to me once already and as it was within warranty I got a free replacement). I didn't lose any data because the drive was only about to fail, but hadn't failed yet when the replacement arrived. After replacing the drive, the raid array automatically reconstructed the drive I took out, working just as expected.

Try it virtually by LinuxRulz · 2004-10-31 00:48 · Score: 1

My advice is: If you are not sure about sofware raid5 do as I did: Try on vmware (virtual computer software). You'll be able to see if your system runs well with little of ram, lot of ram, drive recovery...
You'll be able to test anything and restore images if somethin goes wrong.

Not only SW RAID is partition based by SenorCitizen · 2004-10-31 01:08 · Score: 1

Some hardware RAID controllers can do this too - I run a server with an HP NetRAID 3i (or something) controller with five HDD's. Two are split in one RAID1 partition and one RAID0 partition, the rest are a RAID5 array.

Re:Stick with hardware RAID - don't agree by dusty123 · 2004-10-31 01:11 · Score: 1

If you need a really safe solution, stick with software RAID:

1) Linux SW-RAID is very well tested. It's very unlikely that a software bug will trash your data. HW-RAID Controllers have a firmware that can be buggy. It does not happen often, but if it does you're hosed (dataloss, no solution: closed source).

2) Consider the situation where your RAID-Controller fails. If it's an older model you won't get a replacement easily. No one guarantees that other controllers are compatible with the data format your defective controller wrote on your disks. So maybe all data is lost if you don't get an exact HW replacement.

To my mind HW-RAID has the advantage to be faster (but only with expensive controllers) and does have a battery buffered RAM which prevents dataloss in case of a power failure.

Maybe... by warrax_666 · 2004-10-31 02:35 · Score: 1

Maybe you forgot to change the partition type of the 'old partition' to RAID Autodetect? Or maybe you forgot to change the root= kernel parameter in lilo.conf or grub.conf?

--
HAND.

Re:Maybe... by julesh · 2004-11-01 03:50 · Score: 1

Hmmm... I'd checked that initially, but while looking at it again, I've not set it right on the _new_ partition... that is the one that it is adding to the array properly every time. Perhaps because that isn't set right, it isn't looking for the other ones. I'll have to wait until the system isn't in use to reboot it and find out, though. :)

Don't know whether you're by warrax_666 · 2004-10-31 02:53 · Score: 1

just joking, but being an expert in one area does not automatically make you an expert in another area. (Not that his advice about trusting brand names and doing you own testing wasn't good, I'm just saying...)

--
HAND.

Sun boxes make better file servers by Anonymous Coward · 2004-10-31 02:56 · Score: 0

I'd buy an old Sun box with hot swap drive bays and go with that. The Solaris software mirror has a 10+ year track record of being solid, and the hot swap drives mean you never have to shut down. Enough hardware for a file server can be found pretty cheap used. You can easily find a used E250 with dual CPU, drives, and plenty of memory for under $2K.

If you want to do software RAID5, expect your writes to be pretty slow on any system. Unlike a mirror that offloads all the work to the drives, the RAID5 requires CPU resources. A hardware RAID5 has a big advantage as it offloads everything from the CPU.

I've also set up a Solaris x86 system with the OS mirrored and a secondary boot, so even if one of the OS disk dies, the system can be booted and used normally. Unfortunately, the instructions on how to do that on Intel hardware isn't that good, so it took a bit of fiddling.

on hardware by Anonymous Coward · 2004-10-31 03:16 · Score: 0

First off: be aware that if a drive does suffer a hardware failure, it will likely take the IDE channel as well as the IDE controller itself off-line. So if you put more than one drive on a controller, it will likely take out the whole RAID when it croaks.

Secondly: do not buy Promise controllers, even the non-RAID ones. Do not buy a controller with a promise chipset. Do not pass go. You won't be able to use more than 3 cards in your computer if you do this.

Thirdly: seeing as you've already spent $1000 on your raid setup, consider spending another $1000:

8 drives are HOT. you will want a case that can adequately ventilate them. You'll want one that has a "drive area" up front with fans that suck the air across the drives. You're spending $200 bucks on it. Give up.
Invest in a motherboard with a lot of PCI slots, if you're going to use multiple controllers...
Seriously consider getting a 3ware card....
Yeah, I know, "software raid"... that's nice when you have 4 drives on 2 controllers. but if you're going up to 8, especially with drives of that size, you've already made an investment that a little more money won't hurt. Firstly, the controller's smart enough to not bork when the drive goes tits-up, secondly, it's also smart enought to know beforehand when that happens, and, thirdly, you're getting a controller that works well with linux, and has been rigorously tested in the configuration you'll be using it in. There's no "Oops, this chipset is too stupid to use UDMA if you add a sixth drive" option in the kernel for any 3ware controller.

Also, as a side note, if you've bought PATA drives, for the love of god get rounded cables. Otherwise you migght as well fill your case with insulating foam and set it on fire.

This is even stupider. by dmaxwell · 2004-10-31 03:22 · Score: 1

http://ohlssonvox.8k.com/fdd_raid.htm

Configuration is the key by Anonymous Coward · 2004-10-31 03:25 · Score: 0

I might be too late to be modded up but here goes...

I used Linux software RAID and was very satisfied. Success depends of proper config, not all of which is immediately obvious. Some issues

Use disks from different batches
partition properly
tailor RAID level to purpose (RAID0 for /tmp vs. RAID5 for main storage, RAID0+1 or 7 for database files etc.)
read the docs
- There is a Linux HOWTO that goes in details, read it.

Online raid5 expansion by Anonymous Coward · 2004-10-31 04:12 · Score: 0

You might want to consider what will happend when you want to add a hd to the array. Do you have enough space to copy over all the data when you reconstruct the array? There is a tool to resize an array but it is beta at most. I hade it fail on me and I had to restore from backup.

This is my biggest gripe with linux software raid.

What 4+ channel IDE and SATA controllers.... by wardred · 2004-10-31 04:15 · Score: 1

are out there that are decent, and will work as simple IDE controllers?

I've had difficulty finding anything that has more than two controllers that doesn't force you to use the half-baked raid.

All I'd like is a good ata-133 controller (that's what I have, but others may want to know about sata) that actually has at least 4 IDE chanells that show up as normal IDE to the Linux kernel, rather than forcing me to use their weird raid.

Why software raid... by segfault_0 · 2004-10-31 04:20 · Score: 1

It might be off topic, but it seems you could buy SATA drives and say a 3ware card for just a little bit more and get hardware raid with hot swap (and yes there are linux drivers).

--

I was crazy back when being crazy really meant something. (Charles Manson)

Software vs. Hardware - Landscape has Changed! by Chris+Tyler · 2004-10-31 06:29 · Score: 1

I've been doing RAID in various ways for about a decade. The hardware vs. software landscape is very different than it was a few years ago:

(1) Hardware RAID -- this used to be the obvious choice; why load up the main CPU doing XOR loops? However, we are now seeing a lot of 'raid' cards that are in effect little more than multi-interface cards with some code in a BIOS rom, forcing the CPU to do all the work anyways. Putting them aside -- and those will be most of the low-end cards -- the higher-end hardware RAID cards just don't provide the bang for the buck that they used to. This is because...

(2) Software RAID has improved dramatically, especially in Linux (and I presume but cannot confirm, in the BSD's). The processor instructions that were added to the CPUs to accelerate graphics (3Dnow, SSE, ...) happen to work nicely for RAID calculations as well. This is why you'll see some kernels run through trials to see which of several different parity algorithms is faster on the current CPU. This combined with other tuning done in the RAID code has reduced CPU overhead to who-cares levels.

The current software RAID implementation on Linux does hot sparing and rebuilds quite nicely. I haven't had a drive fail on a software RAID box, but I've simulated it, and the recovery was exactly according to spec. The only standard gotcha is booting when the boot drive has failed -- software RAID is little help there.

The ultimate solution for many systems (imho!) is LVM on top of RAID -- providing flexible partitioning, the ability to migrate to new devices, and snapshot backups in addition to the peace-of-mind of RAID-- and all for less than 3% CPU utilization.

Consider [Soft] Hardware Failures... by Anonymous Coward · 2004-10-31 09:57 · Score: 0

Consider soft failures... What happens when a fan dies? Your system overheats. Your CPU begins miscalculating every nth number.

Granted, N is pretty big. But it just happened to me on my server. Things generally worked right. But 100 md5sum's on a set of RedHat ISO's turned up a 2% error rate. Generally only a couple of bits out of multiple 650 Meg images. But, you know, random file corruption really mucks everything up.

Re:CONFIGURE IT RIGHT!! small parts... by Kashif+Shaikh · 2004-11-01 01:15 · Score: 1

You are trying to work around bad sectors causing total disk failures. MD should create a 'bad sector table' for each drive, and when block 0 is referenced, it will write to block 9999999 instead.

And vice versa for reading. Then you don't need to do the hack you are describing.

Kashif

RAID 6+ by TexNex · 2004-11-01 14:28 · Score: 1

Many have posted about the loss of more than one drive being v. bad. So my question is, does SW RAID support RAID 6 or 7? If not, can you do 10 or 50?

RAID by rhedi_phredi · 2004-11-01 14:41 · Score: 1

Realize that this thread has come and gone, but that makes this comment approriate - RAID systems toward end of life. Dell gives warranties for effectively five years (initial three year period + 2 one year renewals), this means that their equipment actuaries see the writing on the wall rather than unquestioningly accept the 2 million hour numbers spouted by the disk drive manufacturer. Have a replacement strategy in place as you approach the end, factor in how long it will take you to transfer the data that you've accumulated during the years of successfully operation. And as an aside, my biggest problems resulted from the ON-OFF cycle. Implement a source of constantly supplied power as well as some system to actually be able to "hot swap" the drives and the cooling fans, cycling the drives to replace a failed or failing drive may push additional drives to failure.

Use linux "network block device" instead. by Cryptnotic · 2004-11-01 19:57 · Score: 1

NFS doesn't serve partitions, it serves files.

--
My other first post is car post.

Re:Use linux "network block device" instead. by jusdisgi · 2004-11-05 12:09 · Score: 1

Good point. Hadn't thought of that.

--
Given a choice between free speech and free beer, most people will take the beer.
Re:Use linux "network block device" instead. by jovlinger · 2004-11-10 09:09 · Score: 1

network block device.. linux only tho

The fans must be vibration insulated. by Futurepower(R) · 2004-11-04 17:00 · Score: 1

Absolutely true, in my experience.

The fans are not balanced correctly even when they are new! We've bought 50 of one model of fan recently and all the ones we've tried, 25 maybe, are unbalanced.

A fan mounted near or on a hard drive MUST be vibration insulated. We use nylon plastic straps.

Slashdot Mirror

Experiences w/ Software RAID 5 Under Linux?

541 comments