Slashdot Mirror
RAMdisk RAID?
drew_92123 asks: "I've got a friend who does a LOT of video editing but is on a limited budget. He is currently using a raid array and while quite large, it's not as fast as he had hoped. I had an idea and wanted to know if there is a way to make it a reality, so of course I though of all the brilliant minds here. I have about a dozen Pentium II computers with 1GB of RAM. I would like to upgrade them to 2GB and throw in some gigabit NICs and create a 1.9GB RAMdisk on each one. Then I want to use one of the computers to RAID the RAMdisks together to be shared via Samba most likely. They are all 1U systems, with no HDD's, just a 64MB IDE flash disk. Any ideas out there?" Has anyone successfully put together such a system? How well did it work for you and are there any caveats that you would like to share with others who would do the same?
Wouldn't the bottleneck of the NICs be an issue?
You might just try reconfiguring the raid to be Raid 0+1 (Striped and Mirrored) That would give you the redundancy and speedy access. If the RAM is already maxed out on the video workstation it might be more cost affective to get a better motherboard that supports more RAM.
~george
Before you buy a bunch of hardware, set up one ramdisk with a network link and find out what your real-life tranfer bandwidth is. I'll bet that the gain, if any, would not be worth the effort.
"Eve of Destruction", it's not just for old hippies anymore...
you want to fragment all filesystem access into super-tiny 1500 byte ethernet packet access to each of several hosts for a mere 20gb ram "filesystem"?
gig E does not perform that well on a single host. You'll -might- get lower latency than a cheap raid array but the thruput won't be any better.
why waste time with "RAID" anyways? this ram is all ECC ram and is non-persistent so there's no point in tossing that extra computation into the mix to make it worse.
Just remember, he's SCREWED if the power goes out and he hasn't flushed that /huge/ RAMdisk to a real disk.
Ever thought of just upgrading your SCSI controller? You can get RAID controllers that have insane amounts of RAM in them. That might patch up any access issues.
If your concern is extended duration throughput the multiple rack computers with ram *might* be an option, but most normal users wouldn't consider it due to the latency involved with going through the southbridge then the nic then the nic then the southbridge (of the other computer) then the north bridge. And that's just a one way trip.
Just don't shell out a bunch of money before you do a proof of concept.
~george
At first glance, this sounds like an incredible waste of time. RAID RAMDisk? Why? Are you crazy? What's the point?
But, if you give it some thought it is an interesting idea. Basically you are trying to build a clustered RAM disk..
There is however, a major drawback to this idea. The whole advantage of a RAM disk is speed/performance. Locally, the RAM disk is MUCH faster than a normal disk drive. But, the problem arises when you connect your "RAID RAM disk". You must network the machines in order for them to communicate with each other and suddenly, your performance has dropped to nothing. In fact is is below the performance of a normal disk drive.
In order for your RAID RAM disk to perform equally with a good disk drive you would require a switched gigabit network between your nodes. This will cost more than the "normal" disk. Additionally, even with a switched gigabit network the performance is highly unlikely to exceed the performance of highend disk drives.
So, when you get right down to it, the RAID RAM disk is an interesting idea, just to see if you can do it. But, there isn't really any advantage to it.
1 GBit won't even be saturated by the client: Theoretically 32bit PCI at 33MHz tops out at 133MByte/s, but in reality half that throughput is more like it. The problem is not the throughput. Modern harddisks in a stripe set can just as easily drown you in whatever the PCI bus can handle. Harddisks have high latency compared to RAM, but RAM over Samba over a network has higher latency than harddisks.
Assuming that you'd have to buy at least some of the Gb networking hardware (switches, cables, etc.), you're really not going to be saving much. Assuming at least $100 per 'RAMdisk server', you'll be spending $1200+ for a ~20GB RAID array that will lose everything the minute the power blinks, not to mention drawing several kilowatts of AC.
On the other hand, if you just throw four 100GB ATA-100 drives in a standard tower case with a decent IDE RAID controller, you get five times as much storage for probably about half the money.
Also, remember that most low-to-mid-range PCs can't actually fully take advantage of a gigabit network link, since the PCI bus and CPU get saturated long before the network does.
Um, aren't the network latency and bandwidth constraints going to obliterate any benefit you get from using RAM disks?
and they say the /. editors don't troll much.
Sitting Walrus Blog
[FlameSuit On] I'm sure you all will flame me for this but, I can take it.{/FlameSuit Off]
It is very likely that he is already using IDE or ATA disks, and that is part of his problem. When large amounts of data need to be transferred quickly, SCSI is what you need. There is nothing faster than 15,000 RPM SCSI drives connected to good RAID controllers that have large amounts of cache RAM. Nothing.
If you want high performance then you must use high performance gear. Yes, it does cost 5 to 10 times more than the IDE RAID solution but, there is a VERY good reason for that.
Ok, now comes the flames from the know-it-all masses who's experience is limited to home PCs and no-traffic webservers.
How about a be- *bang!*
*smack* *thump*
*mass cheering*
Btw, it does seem to be a (disturbing) recent trend at Slashdot to try to troll whole stories, instead of just trolling comments. C'mon anyone who's taken even one networking or hardware class knows the speed heirarchy:
cache > memory > disk > network
And, with the amount of physical RAM drives out there (very few), you'd quickly realize that even a local RAM drive doesn't offer enough of a speed benefit to offset it's cost. C'mon editors, I know it sounds cool, but do you really have to post it?
Kurdt
I'm not anti-social. Just pro-technology.
Though what your suggesting would work, and I've done similar things before,
think of the additional costs:
12GB of Ram
12 Gig nic cards
1 >12 port Gig ethernet switch.
Setup time
For what your looking at spending, it may be the same cost as buying some U320
scsi disks and some sort of SCSI raid card.
Sounds like it nbd may be your ticket if you are using Linux. nbd is designed to take a block device, like a hard drive and make it available over a network on a different host. It will also do RAID 0,1,5. Perhaps it will work with a ramdisk. I can't swear that this will work but is sure might, since after all, a ramdisk is implemented as a block device.
RAM is cheap. If you are unconcerned about high electricity costs and need a large *F*A*S*T* device for storage, stripping a number of ramdisks could be the thing to do. PC133 1GB DIMMs are currently about US$200 and are on their way down. Sure, it's expensive compared to RAID 5, but I'm sure it's a lot faster. Just make sure you write out anything you need prior to downing the whole array.
I am not sure if you thought of it, but with that many systems running et, and if you are using mode 0, a crash on any machine would cause the whole array to be unusable. A simple reboot would force 100% data loss.
Buy the RAM and use it with a few of these solid state disks. 4GB per PCI slot. But don't be disappointed if it still isn't as fast as you want it to be: The disks are probably not the bottleneck. I'd be surprised if a properly configured RAID array couldn't deliver adequate performance for video editing. Even single disks are fast enough to work with uncompressed video these days.
whilst the RAM and NICs will not be overly expensive the real cost will be in the 12 port gig switcher. Doing some large scale networking recently I've had occasion to handle a few from a couple of suppliers and the decision we had to make came down to whether we really wanted gig or could live with 100m and 3 or 4 extra servers.
All things being equal I'd put the value of the switch higher than the entire setup as it presently exists...
It's not that I'm Anti-American - I'm Pro-Freedom
Dumb idea.
The idea of a drive is persistant storage. Disk caching algorithms nowadays are excellent and normally surpass ram drives, since in reality, you are pretty much ALWAYS using a ramdrive. Not to mention the network issues brought up in other threads, and the simple fact that the main reason for raid 5 is redundancy of storage in order to boost reliability not just performance, and by putting everything in ram, there goes reliability. Sounds like, if you are not concerned with reliability as much, is a Raid-1 array, which implements striping but not parity.
...about a dozen Pentium II computers...
...upgrade them to 2GB...
Let's see, you want to buy 12 GB RAM, 12 gigabit NICs, and a gigabit switch to get a 24 GB logical volume?
The cheapest gigabit NIC on pricewatch is $40. I'm not sure what type of ram a P-II takes, but everything on pricewatch is over or about $100/GB. So that's $140 per computer, times 12 computers is $1,680 for a 24 GB volume. (This is what you consider a 'limited budget'?)
And you said your friend already has a raid array? I'm willing to bet that it's bigger than 24 GB, and since it's probably attached locally instead of through the network, a hell of a lot faster.
For comparison, you can buy a 250 GB EIDE drive for $325. I'm sure you could put together a cheap computer with four of these drives for less than the $1600.
I have about a dozen Pentium II computers with 1GB of RAM. I would like to upgrade them to 2GB and throw in some gigabit NICs and create a 1.9GB RAMdisk on each one
:)
Are you on a limited budget, or are you going to spend a couple grand to upgrade all these ancient machines?
Did you ever consider just taking the money and buying one really fast machine?
A poster above asked, "Why don't you just put all that RAM in his/her system, or, if the RAM is maxed out, buy a better motherboard."
Problem is, there's only so much RAM a "normal" motherboard can refer to, before you have to start doing ugly (and expensive, slower) hacks. 2^32 = 4294967296 bytes, or precisely 4 gigabytes.
After that, you're on your own.
So, my question is: Why don't we have "RAM banks" that interface over SCSI, firewire, or even IDE?
If it's firewire, it could even be external.
A RAM chip isn't that large. Why isn't there a slick "drive" about the size of an iBook that holds 40-60 modules.
The RAM itself, according to this week's memory prices is $80 for 512 MB PCC133 ECC, which means 5 gigabytes (more than enough probably for whatever work you want to do -- remember, this is volatile memory, so you can't store anything permanent on it!) is only $800 for the chips themselves.
I'm not sure I understand why PC800 is about the same price for 256 MB modules, but an order of magnitude more for 512 MB, while PC3200 (apparently the fastest of the lot?) is almost the same price as what I quoted for PC133.
Anyone?
This isn't a complete answer, for sure, but for linux RAID you need to present the RAM on computers A, B, and C to computer M (the mux) as block devices. You'll probably need to write a device driver for machine M that presents a block interface and speaks a UDP protocol to machines A, B, and C, where your server stores blocks on the local RAM disk according to whatever scheme works for you. Then 'just' edit the raidtab and build your md0 from the block devices. The reason 'just' is in quotes is because who knows if it'll work with a non-disk block device.
Don't forget to deal with lost UDP packets, but you don't even want to go near TCP's latency on this. If you put them all on a switch your packet loss should be negligable anyhow.
I don't think it's practical for your application but it would be a very cool hack. Good luck!
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
I do customer service repair and testing for high end video servers, particularly the RAIDs attached to them. Based on my experience, what you're proposing seems like a waste of time and money. I think your friend would be much better served with a more traditional RAID setup. For single-user editing station a 4 drive IDE RAID-0 should be able to handle the load, and a similar SCSI array should be more than capable. I recommend SCSI.
In typical storage situations you have 2 issues you need to consider: bandwidth and space. With video you add a 3rd issue which can easily eclipse the other 2 in importance: latency. Latency can cause hiccups in your data stream which would be unnoticable in any other application, but become painfully obvious with video, and any networked sollution is going to add latency. The more network is involved, the more latency will be added, which is why I would absolutely advise against a distributed solution. For that reason even if your network has the same theoretical bandwidth as your RAID, it will be slower, and that will kill the video stream.
Anyway, your friends needs should be taken care of with an older (cheaper) SCSI RAID controller and some older SCSI drives, say 9-18GB 7200RPM. In a RAID-0 configuration they should be able to handle simultaneous record and playback of 12Mbps per drive, with a 3 hour capacity at that maximum bandwidth. For example: my test fixture has space for 5 drives, so it can handle 3 hours worth of 60Mbps video, which is decent for HD and ludicrous for SD. You should be able to pull together something like that for less than what you're planning to spend on RAM and Gig-E network gear, and will be more reliable (minimized data loss in case of power outage) and a hell of a lot cheaper to operate (unless your friend lives in the magical land of free electricity).
Bear in mind that what I've described is the test I use to verify the fitness of our drives, and we use it because we've found that it is more strenuous than any commercially available SCSI test setup. Most new drives are able to handle it, but used drives can be a different story even though they might be perfectly good for any other application. With used drives you may have to drop your expectations to 10 or even 8Mbps per drive, so plan accordingly.
That said, you also want to take a close look at your encoding/decoding hardware, as that can be a source of problems. Don't just look at the hardware specs, either, as all to often the driver capabilities fall far short of what the hardware can theoretically do.
Under capitalism man exploits man. Under communism it's the other way around.
I probably won't work *well* mainly for the reasons posted by others above (power outages, latency, COST COST COST), but I think you should try it anyway, and post your results here because it sounds like a very interesting project that could benefit others. But don't spend any money on it yet. Try it out with 100Mb ethernet first just to see if it works.
Since the questioner is looking at using commodity hardware with a commodity OS using a commodity networking protocol, my gut feeling is that (s)he doesn't have a prayer. It is a cool idea, but latencies are likely to be too high.
The /. dreamers don't need to give up all hope, however. :) There is
relevant work in the academic literature, using specialized hardware and software of course. The work
I'm familiar with is from Hank Levy's group at UW. To sum up, based
on what I remember from a class I took back in '98 from
Mike Feeley
(first author on said paper; also did his PhD thesis on the topic):
The motivating example came from Boeing. They had a bunch of CAD workstations all with lots of RAM (by the standards of the day). However, looking at any nontrivial part of the design required more memory than any single workstation. Paging to disk was S-L-O-W. So why not use the frequently idle memory on the other workstations? The result of the UW work was a sort of global memory management, with paging to remote workstations in the cluster as well as to disk. Using memory on the remote workstations was significantly faster than using the local disk.
So what about latency from the network stack? IIRC (and it has been five years since I talked to Mike about this...) they used myranet. In some sense myranet is basically DMA to remote workstations. One myranet node issues a write request in software, which includes the source address in memory for the data to be copied, a target node in the cluster, and the target memory address on the target node. The myranet hardware on the local workstation does DMA from the source memory location, fires it over fibre to the remote workstation, which dutifully does DMA from the myranet card to the memory locations specified by the sender. This is very fast, but not the stuff traditional general-purpose computing has been made of.
Brian
I hate to be frank, but your solution is equal parts ambitious, elaborate, expensive, unreliable, slow, kludgey, and stupid, with an extra helping of stupid.
Buy a single SCSI RAID card with three channels, three 36GB U160 drives (10 or 15K, doesn't really matter), and set up a hardware RAID 0 stripe. You'll save money and be able to edit any amount of video you want. Hell, buy a SINGLE 72 gig 10K drive and a high-quality single-channel SCSI controller. You'll save even more money.
This is the best way to do this. You've at least proven there's at least one other way to do it.
- A.P.
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
FCAL controller: $100
36G FCAL disk (10k RPM): $125
cabling: $50
misc parts: $50
enclosure: $75
for say $500, you could have a 36G RAID-0 array setup, local to the machine, and very, very speedy.
What if all of the 'RamDiskStations' had 100 megabit NICs connected to a switch and the actual work machine had a gigabit NIC connected to the one gigabit uplink on the switch?
...
... and for this to work you would need to find a way to spread the load across all the servers to get your peak throughput.
If it was a 12+ port 100MHz switch with a single gigabit port he could get a peak theoretical sustained throughput of 125 megabytes per second. I am guessing he already has fast NICs in the machines so it is a matter of upgrading the master box to a gigabit NIC (about $90 for an Intel card) and the switch with single gigabit port on it. There is a Accelar 1050 on eBay, 12 10/100 ports and a gigabit uplink for around $50 with 14 hours to go
Note : all NICs are not created equal, use SMC, Intel, 3Com or other quality brand name NICs for the highest throughput (NE2000 cards and clone cards run about half as fast as good cards.)
Of course once you do this you have a dozen 2G virtual drives but they are all different drive letters and you can't work on one big 24G file at once
I suggest addressing the issue. Look at some of the recent versions of the American Megatrends (AMI) MegaRAID card and couple it with a BUNCH of RAM configured 50/50 read cache / writeback cache and some fast hard drives. Used drives are wicked cheap now, and with 5 or 6 9G drives you could could make a nice RAID 5 setup with a spare for when one craters. I think Dell OEMs these as their PERC cards, and I think they are up to PERC/4 now but a PERC/3 is the one you might find used for a semi-reasonable price.
Glonoinha the MebiByte Slayer
Holy crap - those are NICE! If only there was a way to merge a bunch of these cards into one big (massive) drive or to RAID 0 them into a big drive ...
I hope all that power is used for good, not evil.
Glonoinha the MebiByte Slayer
Plain and simply. :-)
This has good ideas in it....
If you have a spare server, using one as a SAN
might be a good idea. Then use a fibre connection
between the two (i.e. no switch! they are about £10,000-£30,000) and a few scsi cards with hd's
you could build a kick ass SAN storage
JNI do a cool 64-bit PCI-to-Fibre that is supported on linux.
You would still be looking at about £5,000-£7,000
for a nice sized SAN, but it will be about as fast as you can get.
Unless you are going to go top end using RJ45 gig ethernet is not really worth it, due to the collisions on RJ45, you only get 20% of the gbit/sec with fibre this is much higher.
..one or two of these :-)
Briefly said: Kicks any RAID (SCSI or not) and your RAMdisk solution up and down the street.
It could be a tad pricey though, as you might wanna suspect.
We suffer more in our imagination than in reality. - Seneca
Your best solution for this setup would be one of the better SCSI cards or a RAID controller if you can afford it and get AS MANY drives as you can stuff into the box that is going to house this. Go with 9Gig drives if you dont need the space. 18Gig 15K would be best and setup a RAID 0 or 0+1 if you can. The more spindles in the setup the better off you'll be.
Duke
FreeBSD: Nothing runs like a daemon with a pitch fork.
Sorry to be one more person to not answer your question directly (does anyone ever on ask slashdot?), but with the volitility of RAM and the cost of commodity hardware spread across 12 machines, it does seem a little bit unlikely to be the most cost-efficient solution.
If I am to understand you correctly, he needs about 24GB to store his files... but perhaps he doesn't need to access all 24GB at once?
If you want to see if it will make an improvement, why not setup a GB ethernet samba connection to one of those P2's setup for a ramdisk? Then you can decide if it is worth it to him, pricewise, to do such a major upgrade before going all out and spending several thousand dollars on equipment. (unfortunately, final latency will probably be about twice what you experience on the single setup, as you are adding a host controller) Perhaps you could eek out some performance gains by copying the files to a local ramdisk, then editing (silly, though, as it should be copied to ram anyway for editing), or using the remote RAM as a scratch disk, so as to maximize the available ram on the host board.
And, of course, there is the list of upgrades that might be more inexpensive, such as - Dual CPU's? More optimized VidCard? Turning off fontsmoothing?
Hardware may just not be fast enough yet. There must be other people here that remembers when running a Photoshop filter meant a coffee break.
The ______ Agenda
The University of Kentucky's KLAT2 project used a FNN to get insane bandwidth without worrying about gigabit cards and switches. I'd suggest you take a look at it.
09f911029d74e35bd84156c5635688c0
1) upgrade drives to WD new 10,000RPM SATA drives, get a number of them and a TRUE hardware SATA RAID card with at least the ability to have 8MB cache/drive via at least pc100 sdr. Eight of these drives with a true hardware SATA RAID card could give you a near fully PCI load(133Mb/s, which is as good as your going to get without moving up to 64/66 PCI system(528Mb/s). The Hardware SATA RAID controller keeps you CPU usage to a minimum(SCSI levels or BETTER) and also gives you the thouroughput needed at 150Mb/s/drive max.
2)look into one of the SCSI LAN systems, using a SCSI160 or 320 controller to link computers together, and look at building some inexpensive machines like duron800/k7s5a combos for 100$ + 1.5Gb of ram per. The SCSI 160 connection easily fills up the PCI bus on the RAM machines and gives pretty good thouroughput because SCSI is very low latency. On the HOST machine, you can use pci64/66 so your BUS isn't much of a bottleneck. then you can use Linux Ramdisk on the RAM machines and export the drive over the SCSI interfact and mount it on the host. You can then create a RAID array on these if you like, but i think its unnecessary and just piling the RAM drive mounts together in a lateral array would be faster considering that RAID on the mounted disks will be in software, and lateral spanning is less processor intesive than striping
You like being, frank.
FYI, professional video editors only format a small part of their disks for RAID striping and leave the rest unused. Depends on the drives this could be 30-50% used and 50-70% unused. IIRC more densely packed blocks at the head of a disk means that the read/write head has to move less distance to do the same amount of work. Ie, it can do more faster. You might have your friend try that. Also how much data is he trying to read/write at once? You SCSI card and motherboard make a big difference here. Does the SCSI controller need a a 64bit PCI slot? How about a 64bit/66Mhz slot? Even better how about a PCI-X slot (64bit/133Mhz)? Most of the dual channel u/320 SCSI controllers want 64/66. Also how does the mobo handle the bus connection to the slot(s) your SCSI controller(s) is (are) in? Does it have to share the bus with anything else or is it a dedicated bus? Different mobos do different things. Usually the higherend mobos put the 32/33 PCI slots on one bus and either the 64/66 slots share a bus or they have seperate buses. The mobos I've seen with PCI-X had their own bus. Do they share the bus with the AGP slot? Another important thing. Is you friend writing using the network during these editing sessions? A typcial copper GigE wants 32/33 or 64/33. The fiber cards (which are usually much more higher end) want 64/66. Another thing, Windows software RAID SUCKS. It really is horrible. Linux software RAID is so much more unbelievably better. If he's doing video editing,though, he should most definitely be doing hardware RAID. Don't skip on the controller. LSI makes some very nice controllers. Adaptec's are nice as well (they are like Sony. The make a nice product but it never wins the benchmark tests. It's a solid performer though). Do no use onboard OEM ATA RAID (you didn't specify ATA or SCSI). It sucks as well. I have in my Asus A7V333 w/ RAID the Promise OEM FastTrack133 controller and it causes CPU spikes every minute on the dot. Get a real hardware controller. Enough of my ranting. Good luck!
just get an SGI octane2 and be done with it.
seriously though, just stripe a bunch of plain ata drives (2-4) I use raid 0 all the time for video editing (even uncompressed video) and my 2 drive setup is quite speedy(although the 4 drive array is nice too).
Check out the SAMSON project. They try to do stuff with "memory servers" that lets one computer access the RAM of another through a network.
so lets say you are going 5 horus a day 200 days a year. thats 1000 hours a year, or 1000 kilowatt hours,, or 1000* 10 cents for 10,000 cents, or 100 dollars. or 1000 * 20 cents in a high electricity cost area, for a cost of 200 dollars.
nice fucking budget ya got there. go buy a freaking RAID controller that will let you stripe to 4 or 5 discs at once, and get a few HDs with a bigger cache. and more RAM.
what you need can be found here. The enhanced network block device allows you to share a block device, like a ram drive or hard drive, over the network and make it appear on the main machine as a normal hard drive plugged into it. i have created a raid0 over this before. I can't really comment on speed, as it was a 10Mbps network, but it did work. with fast machines and a fast network, i would imagine you would easily saturate gigabit networking.
Why read the article when I can just make up a snap judgement?
One of the big issues with RAID today is that drives are so large it is simple to hit your size requirement and still have a RAID that doesn't perform to the required level. This is because most people spec a RAID to have a required size and assume it will meet their performance requirements. Since the performance of a RAID is directly related to the number of drives used in the RAID, they should either use a larger number of smaller capacity drives or use the large capacity drives, but use more of them, even if it exceeds the space requirements for the RAID. Look at any review of RAID controllers and they will test with 2, 4, 8 etc drives and you can see the performance increase with each added set of drives. Even RAID 5, which requires more calculations for each drive added, will only slow the rate of growth not stop it.
"Computer Scientists can count to 1024 on their fingers" (non-mutant, non-mutilatated, human computer scientists)
To answer your question (redundantly, I guess) use nbd. (network block device). Linux-specific solution.
But you don't want to do that (for reasons posted elsewhere in this discussion).
You refer to this as "a raid array". You don't supply more details, which makes me suspect that your raid array isn't well understood. Which means you perhaps have RAID 5 set up, perhaps using IDE RAID and perhaps have multiple disks per IDE channel.
What you need (IDE or SCSI, don't care) is multiple disks set up with a RAID 0 configuration for speed. If you need reliability (and I assume you don't, since you are considering RAM disks) it is best added via RAID 1+0 or 0+1.
This introduces mirroring, either by mirroring your disks in pairs and then raid 0'ing those pairs together or by taking two big RAID0 devices and mirroring them. [Nice thought exercise. Why is 1+0 better than 0+1? Hint: what happens if two random disks fail.]
This will half your storage. But you said you had plenty of storage.
Solution:
Reconfigure to a more appropriate raid config (1+0, 0+1) and ensure you aren't using a really cheesy solution.
(DON'T put multiple disks on the same channel with IDE RAID! Performance will suck as the disks contend for the channel. You may say that you have an IDE raid controller with two channels, max of 4 disks and only by putting two disks on each channel. That is why those controllers suck. They are OK for mirroring. If that is the case, you will be faster overall by using software RAID over 4 disks on your 4 IDE channels (2 from your mobo, 2 from your IDE RAID solution).
Thanks for the off-the-wall idea though. Caused us all to think a bit.
Just buy and EMC Symmetrix or HDS 9960 and load it with 64GB of cache. ... oh wait, we're living in a post-dot-com reality now.
Nevermind.
I bought 10 Fibre drives, 18gig 4meg buffer.
10 FCAL Adapters
1 hub
2 Optical Links
2 Optical cables (want to put them in the basement where it's quieter)
1 PSU / external case
Total cost so far is around 500$ for close to 270 gig of storage with 1 gbps speed. I'll post some benchmarks if you are serious about wanting to do this.
Try this out: http://www.superssd.com We put a big database on one of these babies and cut I/O wait down by 1/100 of the time.
I do amateur video editing myself, and I recently bought a new system for ~$600 USD for that purpose. They system is IDE RAID w/ AMD 1700 and 512mb of PC2700 ram. I use 2 WD 8mb Buffer 80gb drives in RAID-0 and my throughput for that is enough to record at full HALF FRAME rates (~60fps) AND surf the web (without dropping frames). If your friend is having problems with RAID performance, perhaps they should dump thier RAID card and buy the Epox 8k5a board with dual raid... (its a hell of a lot cheaper than a cluster too)
What I personally would like to see is a resurgance of RAMdisk cards. Full-length PCI cards chock full of DIMM sockets, or even SIMM sockets to make use of all that old memory you have hanging around.
:)
Have an IDE or SCSI socket on the top edge (or a SCSI connector on the back plate) to plug it directly into your disk system. Looks like a disk, acts like a disk, runs like a demon. I'd like to have my swap file on it.
The machine I built last has PC4200 Dual Channel RAMBUS Memory.
With the built-in Promise IDE RAID, I get >40MB/S thruput.
That over 100 frames/sec encoding DiVX.
Sounds like your bud's problem isn't in hard drive speed/thruput.
Truth isn't Truth - Guliani
Try firewire drivers, supposesly they get faster data transfers than IDE, even the lastest...
This has to be one of the stupidest questions ever asked on Slashdot, and only a completely retarded editor would accept it. So the SCSI transfer rate isn't enough for you, so you want to replace it with an ETHERNET connection? Oh boy...
Notes pg_dump has a few limitations. The limitations mostly stem from difficulty in extracting certain meta-information from the system catalogs. * pg_dump does not understand partial indices. The reason is the same as above; partial index predicates are stored as plans. * pg_dump does not handle large objects. Large objects are ignored and must be dealt with manually. * When doing a data only dump, pg_dump emits queries to disable triggers on user tables before inserting the data and queries to re-enable them after the data has been inserted. If the restore is stopped in the middle, the system catalogs may be left in the wrong state.
This sounds more interesting as a distributed computing project, rather than for mere video editing. My personal experience is the typical 100 gig IDE drives they're selling now are plenty fast enough for DV editing; the bottleneck is in processor speed, for effects, MPEG encoding, etc.
OTOH I've sometimes thought, in OO information systems, if you never had to persist data to disk, it would sure save a lot of trouble. Multiply-redundant storage of data in memory on lots of machines on a network, with separate UPSs, might alleviate the need to save it to disk. Only in certain applications, of course.