Slashdot Mirror

I made a Fibre Channel array like this last year. The only difference being that I used a Mylex eXtremeRAID 3000 (eBay for $200), a 256MB Crucial DIMM for cache, and four Seagate ST39102FC 9GB 10,000 RPM disks.

My whole point to the project was EXTREMELY fast disk access (up to ~160MB/sec sustained transfers, see here) that I could locate at the far end of a REALLY long cable. I've got my machine in my office and the hard drives on the other end of a 30m cable, nestled nicely down in the basement where I cannot hear it.

There are a few basic pictures of the external assembly available here. Works really, really well. It's amazing what hugely fast disk IO does for the rest of a machine.

I made a Fibre Channel array like this last year. The only difference being that I used a Mylex eXtremeRAID 3000 (eBay for $200), a 256MB Crucial DIMM for cache, and four Seagate ST39102FC 9GB 10,000 RPM disks.

My whole point to the project was EXTREMELY fast disk access (up to ~160MB/sec sustained transfers, see here) that I could locate at the far end of a REALLY long cable. I've got my machine in my office and the hard drives on the other end of a 30m cable, nestled nicely down in the basement where I cannot hear it.

There are a few basic pictures of the external assembly available here. Works really, really well. It's amazing what hugely fast disk IO does for the rest of a machine.

I wonder if it would be possible to use a PCMCIA -> PCI Card adapter with a normal SCSI RAID card and two of those 15k drives to complete the expirence. Note: This is a joke! (In theroy it should still work though!

I hunted all over the net looking for benchmarks of SCSI systems versus IDE systems, and couldn't find a damn thing. Sure, people benchmark SCSI hard drives versus IDE hard drives, but nobody ever bothers to benchmark SCSI RAID versus IDE RAID.

I got so sick of it I said what the hell, and ordered up a pair of raid arrays for doing my own tests.

Test system configuration:
Supermicro P4DP6 Mainboard with two Intel Xeon 2.2GHz processors, and 4GB of memory.
Microsoft Windows 2000 Server

No I did not have time to test this under Linux before I had to get these bad boys ready for prouction. I doubt the benchmark results would have been much different, I've seen 3Ware running under Linux at Linuxworld and they kick some serious ass.

IDE setup:
3Ware Escalade 7450 Raid Controller
Three Seagate Barracuda ATA IV Drives (7200RPM 9.5ms access time)
This was set up in a RAID5 configuration.

SCSI Setup:
Mylex AcceleRAID 352 Raid Controller
Three Seagate Cheetah LVD160 Drives (15,000RPM 3.6ms access time)
This was also set up in a RAID5 configuration.

Benchmarking utility used:
HD Tach 2.52

Here's the bottom line I got out of my benchmark tests.

SCSI Performance
CPU Usage: 2.1%, Access time 6.1ms
Read Speed: Max 37.6MB/s Min 11.6MB/s Avg 30.8MB/s
Write Speed: Max 8.5MB/S Min 5.4 MB/s Avg 7.5 MB/s

IDE Performance
CPU Usage: 3.1%, Access time 14.2ms
Read Speed: Max 31.8MB/s Min 4.3 MB/S Avg 18.3MB/s
Write Speed: Max 48.7MB/s Min 12.3MB/s Avg 36.4MB/s

I was a bit shcoked to see the IDE do so well. It annihilated the SCSI in terms of sheer write performance but lagged behind a bit from the read performance. CPU use isn't a factor for most people, who really cares if you lose 1% more of your CPU to the IDE compared to the SCSI.

Those 15,000 RPM drives were loud as jet engines, and they got hot enough that I was thinking of cooking some bacon strips on them. They were too hot to touch. The IDE drives on the other hand were barely audible even with the case off, and remained completely cool to the touch through the whole test without even a fan on them. You tell me which of those two types of drives is going to have a longer MTBF...

I didn't even use high performance IDE drives for that test. I'd also like to point out that the Mylex card was 66MHz/64bit, whereas the 3Ware card was 33MHz/64bit... so the 3Ware card was holding its own even though it was running at a slower rate of speed. I wonder what will happen in future generations of these controllers when they turn up the speed and improve the code...

Cost... I coul have built three of those IDE Raid systems for the price of that one SCSI system.

Space... The IDE drives were 80GB, the SCSI were 36GB. IDE owns SCSI in terms of space. We have some bigass databases where I work so that's actually fairly important to us.

Unless you REALLY need that 6.1ms access time or the extra ~20% in read performance you are far, far better off with an IDE Raid at this point.

The guys at Toms and Anandtech really need to do a major article on this stuff...

For the skeptical, here's a link to the screenshots of the HDTach benchmarks I ran. Be GENTLE guys we do not have tons of bandwidth for this...

IDE vs SCSI

IDE is on the top, SCSI is on the bottom. Interesting how SCSI is fairly linear but IDE is really sloppy and just running all over the place.

Now, I'm not going to claim to be a RAID expert, I'm just a beginner, but it's my understanding that RAID 10 is NOT more reliable/redundant than RAID 0+1.

Mylex agrees that in a RAID 0+1, up to TWO drives can go out, as long as they are non-adjacent members of a ray. How RAID 10 could be more redundant, I don't know.

10 machines with SANArray FFx-2 from Mylex - that's up to 9TB per machine... It writes up to 270MB/s...

Set up a Round Robin DNS system - and write a small application to run on the servers that makes the DNS point to the next machine when it's full...

Put a DVDWriter on each machine and viola ;)

> and if you're into scsi, buy one of these raid-
> controllers with memory-with-battery-support on
> them.

Mylex builds them.

First off, I think you've made a number of incorrect assumptions. My views are based on years of corporate experience, including PC rollouts. Please read my responses below. Understand that I am the only person who gave you an useable, DOS/real-mode solution. And it's not as expense as you think.

SCSI is not that cheap! Perhaps for a home system, but my company is betting it's business on the systems that we buy. That means quality, reliability, and driver issues are a big deal to us.

So are mine! You think I've been fired for buying SCSI all these years? More $$$ does NOT equal quality. I go through specific products below ... (and note that NONE say "Adaptec" -- been burnt by their crap too many times).

Each change in a driver results in a different build of the OS image. If we use a no-name SCSI card, each time the support chipset changes we need to build a new image. This is very expensive for us to maintain.

All of the cards I use have quite stable drivers. Of course when you buy something new, you shouldn't expect it to work. You should always wait ~6 months for the bugs to clear out. But when if you'd waited 5 years for good Adaptec Linux drivers, then you'd get quite irritated.

You can easily standardize on one SCSI chipset, the TekRam TRM-S1040:

• Low-cost end-user boards, TekRam DC-3x5U/UW series:
  • $15-20 UltraSCSI TekRam DC-315U for internal/external SCSI peripherials (no BIOS) -- This is probably all you need!. Much faster, cheaper, better and more compatible than Adaptec's AIC-7850-powered 2906
  • ~$40 UltraSCSI TekRam DC-395U for booting devices (BIOS) -- cheaper and better than Adaptec's AIC-7880-powered 2930 IMHO
  • ~$60 UltraWide TekRam DC-395UW for 40MBps Wide devices (BIOS)
• Single driver for all boards in series
• Excellent, direct vendor cross-OS support, DOS, 9x, NT/2000, Linux, *BSD, Solaris, SCO, NetWare, BeOS -- including full boot disks for just about any flavor. Check them all out -- especially Linux, *BSD or BeOS users, never seen such support!
• Although the chipset is just over one year old, I have seen 0 issues with drivers since March of 2000.

We cannot afford to put a Zip, Jaz, CD-R/RW and DVD-RAM/RW drive on every PC in my office. Instead, we have one or more external ones and put a $15-20 TekRam DC-315U in each system. Works great! Also great for cloning when I don't want to hit my server/network too hard (in the middle of the day), let alone transfer loads of data between systems. In Linux, I can even load/unload the TekRam S1040 driver on-the-fly, flipping drives on/off various systems without a reboot/shutdown. It's _awesome_ bay-bee!

As far as other experiences, I recently had to chuck my Adaptec AHA-2940UW (AIC-7880) in my Linux server because it is a POS (in 6 years of using Adaptec on Linux, I have yet to have a good experience thanx to their non-direct support). The sucker refused to work properly with a new, $4,000 Exabyte Mammoth2 60/150GB tape drive (talk about "betting my company's business" on a SCSI card!). I replaced it with an $60 Advansys (now owned by ConnectCom) chipset-based card:

• $60 UltraWide SIIG AP-40 Pro -- also readily available at your local computer/electronics store (although you'll pay about $99 retail).
• Advansys is known for their excellent direct driver development, and broad OS support (first vendor to officially support Linux -- way back in 1995)
• Has full per-device configuration in BIOS, just like Adaptec (i.e. Ctrl-A at boot). Works much better and more compatible with more devices than Adaptec IMHO!

But if you need faster still, Symbios Logic (now owned by LSI Logic) is always faster and more ubiquious than Adaptec. So much so that Adaptec attempted to buy Symbios out (since they were kicking Adaptec's butt in the OEM and FibreChannel market). You'll be interested in the popular 53c895/1010-series:

• Mid-cost, end-user boards in the TekRam DC-390U2 series -- 53c895 Ultra2/LVD (aka Ultra80) chipset:
  • $100 TekRam DC-390U2B for single channel Ultra80/LVD (or UltraWide) channel
  • $130 TekRam DC-390U2W for single channel Ultra80/LVD and isolated UltraWide bus
• Dual-channel, 32/64-bit PCI end-user boards in the TekRam DC-390U3 series -- 53c1010 Ultra160/LVD chipset:
  • $175 TekRam DC-390U2W for single channel Ultra160/LVD (or UltraWide) plus single channel UltraWide legacy
  • $235 TekRam DC-390U2D for dual-channel Ultra160/LVD (or UltraWide)
• Symbios Logic 53c8xx-series supported natively in just about every OS -- many chipset are upward compatible (with exception of 53c1010 that requires a new driver -- but still better than Adaptec's cards, especially their newer ones)
• Better than Adaptec performance at any chipset/protocol (usually by an average of 5-10%)
• Widely supported, numerous OEMs, >10 year-old 8xx-series design/support
• The best damn cabling/converter bundle I've ever seen in a kit (boy is Adaptec stingy!)

And when it comes to hardware RAID, Adaptec is just NT/Netware-only. As such, I prefer DPT or, better yet, StrongArm ASIC-powered Mylex RAID controllers with broad OS support (and better performance too).

So what brand are you blindly putting your faith in? Eh?

SCSI hard disks are much more expensive than IDE. I just checked pricewatch, and a roughly equivalent SCSI drive was around $200 more than it's EIDE counterpart (36GB)

And those IDE drives can be put in a $20-40 enclosure and made to work at 20MBps+, right? Not! When it comes to external (isn't that what we are talking about, eh?), IDE is a joke -- with slow as molassas USB (even in 12Mbps/1.5MBps "fast" mode) being the only option (although new ATAPI-to-FireWire bridges, like this Ultra33 one from Intito, is changing that -- although it requires OEM firmware/programming). Plus we're back to the DOS/real-mode issue (even for FireWire). Only SCSI is "ready-to-go" external.

Now you can compare GB/$ all you want. You do NOT need the latest SCSI drives. Go with a late-model 9-18GB SCSI drive. I mean, how much storage do you need? We're only talking $100-200 for the drive, another $20-40 for the enclosure and another $10-30 for cabling and termination, max. You could do it for under $150, including cables and termination, if you pinch your pennies (and buy your stuff mail-order -- use Cyberguys for SCSI cables/terminators). Plus, you must be looking at 7,200-10,000rpm RPM drives -- don't make the mistake of comparing 5,400rpm IDE drives to obviously much faster SCSI drives.

-- Bryan "TheBS" Smith

To Whom It May Concern:

This post comes late in this discussion, and as such I have no guarantees that you'll end up seeing this message. However, your group has made a decidedly honorable request to Slashdot, so I'll take my chances and provide you with some commentary on your acquisition plan:

Motherboard:
Intel® 840 chipset
Integrated dual ATA-66 EIDE controllers


You're dooming yourself to, at best! Abit motherboards. It's tragic, but while Abit motherboards make for excellent gaming machines, they're simply not of Asus caliber. I'm dead serious: Recently, my friends and I spent upwards of 50 hours combined trying to get Linux to work with an Asus BH6 and a large Maxtor UDMA-66 drive. The number of incompatibilies and instabilities I encountered was pretty much the worst experience I've had with off the shelf stuff, bar none.

The Asus just worked, and has been rock solid for me ever since.

In any system you build, reliability is far more important than speed, and if you poke around you'll find far, far, far fewer horror stories referencing Asus motherboards than any other brand. The reason is simple--if they can't do a technology right, they won't do it at all. Asus has not yet released a motherboard with integrated UDMA-66 for this reason, and they're pretty much the hardware company I respect the most because of it.

If Asus hasn't released a RIMM supporting Mobo(they may have, I don't know), there's a reason.

CPUwise, has Asus given in on releasing a K7 mobo yet? Last I checked, Intel was playing off people like me who won't get a motherboard unless Asus makes it by threatening Asus with extreme price increases if they supported AMD's chips. Nasty stuff.

You should specify UDMA-66 expansion cards, and leave 'em empty.

SCSI:
Integrated Ultra 160/M SCSI and Ultra/Wide SCSI Controllers
9GB Ultra 160/M SCSI (7200 rpm) hard drive (internal)
36GB (or larger) Ultra 160/M SCSI (10000 rpm) hard drive (internal)


SCSI on the motherboard is unnecessary, and you're risking greater downtime through the loss of replaceable components. A spurious shock through the SCSI line can (rarely, but possibly) short out your SCSI bus. Replacing a card is a hell of alot less downtime than replacing the system's motherboard!

Furthermore, it sounds like you plan to store quite a bit of content on this machine--I'd be interested in your design justifications for two, drastically different sized hard drives. Considering the amount of storage you're planning to use, I'm wondering if you shouldn't spec out using Hardware RAID-5 w/ three 36GB drives. That would give you much more aggregate transfer speed, as well as hot-swappable reliability(you'd be able to lose any single drive yet not lose a byte of data). While I understand RAID-5 is much more of a server technology than one you'd expect on a workstation, your workstation has been spec'd with server level design considerations and I can't imagine why the storage solution should be any less professional.

With regards to the controller, I'd normally suggest a solution based upon Adaptec's generally excellent hardware, but Mylex's eXtremeRAID 2000 looks like it'd fit your needs quite nicely, and has company-supported Linux drivers.

Diskwise, I've heard good things about IBM(who invented GMR, the technique a good chunk of the industry depends on for the kind of high density platters we know and love) and Seagate. I'd at minimum specify a range of brands you'd accept for the hard drive--remember, moving parts = more likely to die.

Networking:
3Com 3C905B-TX ethernet card (PCI)

(Disclaimer: I work for Cisco, but this advice long predates my employment there.)

Get a Tulip(DEC 21440 or Compaq-Purchase Remarked Equivalent) based card. Yes, they're inexpensive, but Beowulf code was originally developed on Tulip equipped clusters of machines. As such, Don Becker has optimized their drivers to an absolutely ridiculous degree, and there are several kernel networking settings that are just not easily available for any other architecture. (I believe the Intel cards have some of the fast routing code ported to them.)

In general, Tulips are pretty much the network cards to standardize on, no matter what your operating system.

Multimedia:
Diamond Viper V770 Ultra 32 (AGP 2X/4X)
SoundBlaster PCI128 (PCI)
Powered speakers with wall adapter


Video's OK, I'd suggest something by Diamond based on nVidia's new GeForce256 processor with reservations that I haven't looked into their Linux 2D/3D performance yet. The GeForce256 is a specific model that's likely to end up very well supported, due to its extensive Christmas sales and ostensible inclusion in MS's coming X-Box. (3D Visualization hardware is now completely driven by gamers. Remember when gamers used to get the sims from the Army? ;-)

Sound: Go SB-Live. The Linux drivers for it are excellently stable, the card has digital I/O, and the chipset is likely to become an immensely powerful programmable DSP in the near future. The card also has an excellent noise floor--a striking improvement over the rest of the historically noisy Sound Blaster line.

Specify the four point cambridge soundworks speakers, if you can. One thing you forgot is a microphone! You need one short range, noise/echo cancelling, unidirectional mic that gets mounted on the monitor. You need one long range "speakerphone" omnidirectional mic for conference-over-IP sessions. Both mikes(and probably all sound in general) need to be hooked through an A/V box that defaults to physically separating the microphones from the mic port on the sound card. It's great to be able to use data links as emergency/impromptu voice channels, but you don't want an adversary to use your computer as a listening post!

300W or greater power supply

Ah, power. Hot-swappable power? Not inconceivable, though we're starting to really push the separation between Workstationa and Server with one of those ;-)

Specify a UPS for this machine, preferably one of those "Brick Walls" that can survive direct lightning strikes without sacrificial elements.

Portable Storage:
LS-120 internal superfloppy (IDE)
5X (or faster) DVD drive (internal - IDE or SCSI)


The LS-120's are nice if your organization has standardized on them, but that's about it.

Specify a SCSI DVD drive, simply because you'll weed out the "consumers won't need this more the six months" fly by night hardware makers that only work in IDE. Plus, the CPU load of doing anything in IDE is ridiculous--I did a move from one large IDE drive to another...2.5MB/s, 75% CPU on Celeron 450. Ouch.

Another main advantage of a SCSI drive is that it lends itself well to integration with a SCSI CD-Burner. Don't discount these--there's just literally nothing at all better for moving 2 to 650 megs of data from one machine to another, particularly for emergency drop ships. (I built our groupware CD burning page at work for precisely this reason.) I highly suggest the Yamaha 8x SCSI-3 burner--I just bought one, and much like the Asus, it Just Worked.

Even if your system is prebuilt for you, parts that "Just Work" contribute significantly to the long term life of the system as a whole.

Keyboard:
Soft-touch keyboard (no keyclick)

Specify the Microsoft Intellikey Pro, *NON ELITE*(with the god awful diamond star arrows). Technically, the thing is actually pretty nice, particularly with its feel, but RSI injuries are real and 2000 is pretty much going to be the year of Lawyers vs. Engineers. This is one less thing for you to worry about.

Warranty: 3 year parts and labor

No long term support contract on the operating system? (RH6.x? You might want to replace this with 'Red Hat Linux, Present Revision') I understand why you'd want this--either your in house talent is that good, or you don't want to be stuck with the random low bid being your support provider too. I'd personally vouch for VA Linux as a provider of enterprise level support--stock price aside, these guys know their stuff. IBM and Linuxcare(who doesn't sell machines, but provides top notch 24x7 support) are also good companies.

Other stuff you didn't mention

Modem: Don't try to depend on everything always being up. In a pinch, you need to be able to interact with analog(radio?) communication lines. Definitely v.90, and if it's internal, it must be a full com port implementation. Internal is preferable(nothing to lose), but it'll be harder to find one that isn't a Winmodem. Do not trust the Winmodem drivers for Linux--see the SB Live driver, before they opened it.

TV Adapter--you may need to output to video for presentations or whatnot. This is entirely dependant on your needs.

Temperature--heat sensors are a very, very good thing.

I hope this content was useful. All I ask in return is that if you end up reading this, you notify me so I know I didn't waste my time picking through your acquisition request(which was overall quite good!).

Yours Truly,

Dan Kaminsky
DoxPara Research
http://www.doxpara.com

L8r...
Corrado

read the file describing the configuration.

It's a Siliconrax SR-485 rack mount peripheral chassis with a built in 9" or 10" monitor

It also houses the Mylex DAC960SXI SCSI-SCSI 6 channel RAID controller, w/256MB cache.