High Sustained HD Transfer Rates on a Budget?

aibrahim asks: " I need to be able to sustain at least a 23MB/s (That is MEGABYTES per second) transfer rate over the course of about 2 hours. I'd like to get 60MB/s sustained. I also have to be able to perform seeks quickly because I am going to need random access. Can ATA RAID arrays really do it? What would be the difference between using ATA/66 and ATA/100 arrays? What other budget conscious ways are there to get it done ? How about sharing this fast storage across a network? For some context, the application is non-linear editing of uncompressed standard definition television, multiple streams if possible."

Comments

What about the rest of your hardware?

[bluGill]

In theory a normal PCI bus can reach 132MB/s. However you not only are reading (writing?) that data from your harddrive, I'm assuming that you also need to put that onto your display.

Don't forget cache issues, you DMA that into memory, then read it out to the processor. Can your memory handle that kind of access? Your putting a lot of stress on the memory bus. If your main code doesn't fit into the processor cache (or isn't optomised to fit well) Sure the lastest gigahertz CPUs can deal with the data just fine, but typically PCs can't keep up with the data flow.

For fast disks, SCSI rules. while ATA now allows taged queueing, AFAIK nothing impliments taged queueing in ATA disks, while scsi does this as a matter of course. Meaning that you will want to select disks based on that feature.

Remember, your application is time critical. If a frame is late it matters.

Now can ATA disks keep up? I don't know. Are scsi disks going to be better? Probably. Is the difference enough to matter? Maybe.

In any case, no single disk can keep up with your requirements. What you need is a raid 0+1 so that data can always be read from two disks, in a good implimentation you read from which ever drive is less busy now. Unfortunatly your writing costs go up as you add more drive to make the reading faster. If you can put data on a different disk so that you never read from the same one you write to you will have better luck.

You'll probably...

[Tower]

have to go with a decent SCSI solution on this one... You can't depend on ATA/66 or 100 for any sustained transfer at all. The protocol doesn't support disconnect, so one drive can hold up the channel while you wait for another. Physically, ATA/66 *cannot* sustain 60MB/s.

With 'older' UW SCSI hardware (2 1997/1998 9GB 7200RPM IBM drives) I can sustain ~12MB/s from each, and if I add in my 10krpm drive, I can sustain a total that essentially maxes out my 40MB/s UW SCSI link. If you *need* to keep near 60MB/s, U2W is really your only cost effective choice. Get 4 drives and a card... yeah, it'll run you a little $$$, but you actually will have the performace (striping the disk set, of course).

If you have a dedicated 100BaseT Ethernet link, you might be able to get 20MB/s but not 60... certainly not onto the same system as the drives (PCI 32b/33MHz is ~132MB/s max).

Best of luck.
--