Slashdot Mirror

← Back to Stories (view on slashdot.org)

High Sustained HD Transfer Rates on a Budget?

Posted by Cliff on from the stressing-that-hardware dept.

aibrahim asks: " I need to be able to sustain at least a 23MB/s (That is MEGABYTES per second) transfer rate over the course of about 2 hours. I'd like to get 60MB/s sustained. I also have to be able to perform seeks quickly because I am going to need random access. Can ATA RAID arrays really do it? What would be the difference between using ATA/66 and ATA/100 arrays? What other budget conscious ways are there to get it done ? How about sharing this fast storage across a network? For some context, the application is non-linear editing of uncompressed standard definition television, multiple streams if possible."

8 of 18 comments (clear)

  1. remember the saying... by Wakko+Warner · · Score: 2
    "cheap, fast, good. pick any two."

    If you go with IDE (and software RAID) you've got cheap and fast down.

    SCSI RAID would, of course, be "fast" and "good".

    - A.P.
    --


    "One World, one Web, one Program" - Microsoft promotional ad

    --
    "Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
  2. A traditional PCI mainboard is a bottleneck! by BitMan · · Score: 2

    First off, AVOID:

    • The Promise FastTrak cards
    • Mainboard BIOS RAID (same thing as the crappy Promise cards)
    • Or more than 2 ATA drives in Software RAID if you need your CPU at all (too much CPU utilization).

    Additionally, what you need to consider:

    • PCI Throughput -- try putting your controller on its own PCI channel and go for 66MHz (or PCI64, but you'll need Linux 2.4 for 64-bit PCI) -- you'll need a more costly mainboard to do this, ones based on the ServerWorks (fka RCC) ServerSet chipsets come highly recommended. Otherwise, you'll easily saturate a traditional mainboard with a single PCI bus with a I/O rate that high.
    • SCSI since you're going to need more than 2 stripped drives -- try ~8 Ultra160 drives stripped over 2-4 channels on a 66MHz PCI64 card. The new Mylex eXtremeRAID 2000 is such a 4-channel Ultra160 board with a powerful StrongARM 233MHz at the core (compared to whimpy i960 66-100MHz co-processors on other SCSI RAID controllers).
    • If cost is a factor, look into a "Real" co-processing ATA RAID controller that acts like a SCSI disk/target like those from 3Ware (which has full Linux support). 3Ware's new 6000-series has Ultra66 support and claims >100MBps read speeds and upto 84MBps write speeds (I'm sure that's the 8-channel board with 8 disks stripped ;-).

    -- Bryan "TheBS" Smith

    --
    -- Bryan "TheBS" Smith
    Independent Author, Consultant and Trainer
  3. Even Faster access, 4Gbit! by ajax142 · · Score: 2

    According to this site http://www.iol.unh.edu/training/fc/fc_tutorial.htm l Fibre, or other places called Fiber, Channels could offer,in theory, a speed of over 4 Gigabit a second. I remember an add in some computer mag offering 2 Gig/s Fibre Hard drives... btw...at these speeds real-time tv-to-mpeg writing could be accomplished with no loss, that is if you could get a card to do some real time encoding.

  4. Fibre channel by willis · · Score: 2
    FC is dope... but not cheap. (I work for a fc consulting company right now).

    I remember checking out the prices at pricewatch for 9.1 gb, 7200RPM FC drives -- I don't know how they are so cheap, but some their cost less than $100 with shipping.

    When you starting talking about HBAs (Host bus adapters) the money comes in ... they are around 1,000 bucks. Plus, you've got to get fibre channel hubs to connect the disks, etc... However, the speeds are incredible. Plus, a single card can access something like 16*8 different disks...

    driver support is also a problem... I don't know if linux currently has any good HBA drivers yet...

    Damn, this is an incoherent post. Main point: FC is incredible and incredibly expensive.


    willis.

    --

    there is no thing
    what else could you want?
  5. Check out Medea... by bakes · · Score: 2

    ...here and their videoraid and videorack products. Don't know if they are within your budget, but they specialise in this area.

    bakes
    --

    --
    Ho! Haha! Guard! Turn! Parry! Dodge! Spin! Ha! Thrust!
  6. IDE is faster than we think by toofast · · Score: 3

    On my Athlon 600 system w/ one UDMA 66, hdparm reports 18MB/sec (bah, what's it worth). But after a battery of tests, this same (one) HD was able to sustain 9MB of mixed small/medium/large files, using the standard UDMA controller.

    One lovely alternative is using a UDMA/66 hardware RAID, suck as the Promise RAID controller. Throw in four drives @ RAID 0 and I'm sure you'll get (at least) 35MB/sec sustained.

    Don't forget that 60MB/sec sustained is half the PCI bus theoretical bandwidth, so probably a 64-bit PCI solution would be best.

  7. What about the rest of your hardware? by bluGill · · Score: 4

    In theory a normal PCI bus can reach 132MB/s. However you not only are reading (writing?) that data from your harddrive, I'm assuming that you also need to put that onto your display.

    Don't forget cache issues, you DMA that into memory, then read it out to the processor. Can your memory handle that kind of access? Your putting a lot of stress on the memory bus. If your main code doesn't fit into the processor cache (or isn't optomised to fit well) Sure the lastest gigahertz CPUs can deal with the data just fine, but typically PCs can't keep up with the data flow.

    For fast disks, SCSI rules. while ATA now allows taged queueing, AFAIK nothing impliments taged queueing in ATA disks, while scsi does this as a matter of course. Meaning that you will want to select disks based on that feature.

    Remember, your application is time critical. If a frame is late it matters.

    Now can ATA disks keep up? I don't know. Are scsi disks going to be better? Probably. Is the difference enough to matter? Maybe.

    In any case, no single disk can keep up with your requirements. What you need is a raid 0+1 so that data can always be read from two disks, in a good implimentation you read from which ever drive is less busy now. Unfortunatly your writing costs go up as you add more drive to make the reading faster. If you can put data on a different disk so that you never read from the same one you write to you will have better luck.

  8. You'll probably... by Tower · · Score: 4

    have to go with a decent SCSI solution on this one... You can't depend on ATA/66 or 100 for any sustained transfer at all. The protocol doesn't support disconnect, so one drive can hold up the channel while you wait for another. Physically, ATA/66 *cannot* sustain 60MB/s.

    With 'older' UW SCSI hardware (2 1997/1998 9GB 7200RPM IBM drives) I can sustain ~12MB/s from each, and if I add in my 10krpm drive, I can sustain a total that essentially maxes out my 40MB/s UW SCSI link. If you *need* to keep near 60MB/s, U2W is really your only cost effective choice. Get 4 drives and a card... yeah, it'll run you a little $$$, but you actually will have the performace (striping the disk set, of course).

    If you have a dedicated 100BaseT Ethernet link, you might be able to get 20MB/s but not 60... certainly not onto the same system as the drives (PCI 32b/33MHz is ~132MB/s max).

    Best of luck.
    --

    --
    "It's tough to be bilingual when you get hit in the head."