...is the only way we are going to take advantage of multi-core cpu's and continue to improve our software. Only through purely functional code can you make guarantees about what can be executed simultaneously and let the machine sort it all out. I'm learning Haskell for this very reason.
I spent 9 months in HCMC as the director of software development for a US based company last year. It wasn't my idea. The Vietnamese-American owner of the company who employed me to go thought it was the thing to do. I can tell you that it is NOT the place to outsource any IT tasks such as programming or web design to. Not unless you have some serious government contacts to get you access to the smartest kids out of the state schools (who will still have only minimal programming knowledge and only on Windows).
I couldn't find anyone there who spoke decent english who knew anything about computers. The best I could find were straight out of two year trade school/junior college amature windows jockeys. Linux? Perl? Fat chance! It is still very much a third world country. Software is pirated wildly too. Don't expect employees to obey any sort of NDA. Also note that since people there do not have credit cards, car payments, mortgages, and are already heavily dependent on their families for most things they need they are usually free to leave your company at any time.
At least they have cable modem in HCMC even though it can be a bit unreliable. Exepect a power outage once a month too. Expect theft. I have had motorbikes stolen, cell phones stolen, etc.
And the corruption...oh my god. We paid off everyone and were solicited for payoffs by everyone. My coworker overstayed his visa by a day. They wouldn't let him out of the country! The soldier/immigrations officer/policeman (all the same there) took him into a side room and basically asked him how much money he had on him. $60 worth of the local currency (Vietnamese Dong) and he was free to go. We paid $400 in cash to the customs guy to get them to let our IP phones into the country when the official tax on them was supposed to have been much higher.
And on top of it all they are still very much communist and most are quite brainwashed. It is in a similar vein to North Korea only not as extreme. Americans are lazy people who cheat on their wives and fuck in the streets and cannot be trusted. They do not know about nuclear weapons, don't know the cold war, don't really know anything about the world context in which the Vietnam War happened. Everyone treated me very nicely of course. No anti-Americanism at all as long as someone stood to gain money from me and I was paying in cash. They are always very friendly to tourists and smiling and respectful. Just don't try to date anyone there or talk politics with anyone as you will surely offend. If you hear someone say something about history which you know is patently false just smile and nod.
Suffice it to say the project did not go well. Doomed from the start. At least I had the good sense to bail months before the shit really hit the fan and the whole operation collapsed.
Vietnam is a fun place to visit and I recommend it. I will be going back there again in a couple of weeks for Christmas. I just won't be doing business there again until the business culture changes dramatically.
Carbon Dioxide + Water + Light energy Glucose + Oxygen + Water
Isn't there a way we can do this on a massive scale and remove carbon dioxide from the atmosphere? And I don't mean plant forests because that isn't likely to happen due to the space they require. Synthesize our own chlorophyll and do it much more efficiently than plants can? Or perhaps skip chlorophyll altogether and go with some other means of using light (or even use nuclear power if some other energy source would be more efficient) to implement the carbon reduction reaction in an even simpler way than chlorophyll does it which can allow us to sequester the carbon.
Or perhaps it would be simpler to find some way to store carbon dioxide. But because it is a gas that sounds troublesome.
Looks like the guys at MIT and various people in industry are looking into the problem: http://sequestration.mit.edu/
Ultimately humanity is going to have to take active control over the climate of our planet. Nothing can be left to nature anymore because as part of nature we would destroy ourselves.
The original poster was correct. It is true that MAC addresses are known not to be unique. They SHOULD be unique but they are not. I once received a dozen motherboards all with the same MAC. Intel server-class motherboards. Not prototypes or anything. The board had been in production for nearly a year. They were set up one at a time and deployed into a huge cluster in the datacenter and forgotten about. Tracking down the mac/ip collisions over the next year as machines with the same MAC addresses happened to find themselves on the same network was not fun. So although we can say that MAC's should be the same we cannot unequivocally state that they are.
I'm not saying it is unmaintainable just that nobody may choose to maintain it. Also the mailing list may actually be working. It seems my MTA was picking the lower priority MX which apparently isn't properly configured to accept mail for that domain or something. And finally, as for my company we will probably just migrate off of reiserfs during our upcoming move to a Xen based architecture/server consolidation.
I don't understand what all the doom and gloom is about. Even if Reiser is convicted, it won't destroy open source. I seriously doubt it will even destroy the reiserfs project or even change its name. Do you think the judge will somehow make his code illegal?
The doom and gloom is that reiserfs was a project coded mostly by paid coders in Russia and Hans himself with few other contributors. It is open source in license but not much else.
The other problem is the negative publicity this gets reiserfs. After this I will never again be able to mention reiserfs without hearing comments about murderers. Whether he did it or not won't matter. Every time reiserfs comes up this incident will be mentioned.
Anyone can take over the project, and since it is obviously very popular, undoubtedly there will be plenty of smart people who want to take it over. Maybe there will be a few forks, but as open source has shown, competition is a good thing. You get to choose the best one.
Not just anyone can take over the project. Reiserfs is a very complicated codebase and he made some controversial design decisions. It has had a hard time getting into the kernel.
It is nice to have the fantasy of people being untouchable supercoders and leaders, however the reality is: just about everyone is going to be replaceable, even Linus Torvalds. There may be some geniuses who may need several people to replace them, but they are still replaceable.
Knowledge in someones head is extremely difficult to replace. Linus would be more easily replaceable in the context of the Linux kernel than Hans in the context of reiserfs because there are a lot more independent people contributing to the kernel who understand it. Not many outside the project contribute to reiserfs. The vast majority of the work seems to come from the paid Russian coders. Hopefully SuSe will step in and support it instead of dump it. It seems like they will have to do either one or the other. Time will tell...
It seems SuSe isn't too happy with reiserfs technology anymore and I'm sure this event won't help. So we can probably count them out. Looks like Linspire is their next biggest sponsor. Think they will step up to the plate?
There has been a discussion on the reiserfs mailing list about this also but either I was just banned from it for talking about the code instead of talking about the kids (isn't political correctness a drag?) or they have shut down the list:
: host mx.comstar.ru[83.242.139.27] said: 554
: Client host
rejected: Access denied (in reply to RCPT TO command)
As a person who helped Hans Reiser get some sponsorship in his early days and an early adopter of his filesystem in major corporate use I never would have expected something like this. It's a disaster for him but there isn't much we can do about it at this point aside from debate his innocence based on zero information. So what about Linux? Even if he turns out to be innocent (and I hope he is) the name is tarnished and the filesystem will probably languish. I was expecting reiserfs4 to be an important part of the future of Linux and Free Software's answer to WinFS. Now what will we do? We all know it takes ages, years even, to design, implement, and test a filesystem. XFS, JFS, ext3, etc. are nothing like reiser4 and lack it's capabilities. WinFS will someday be ready and will someday ship. And with this setback for Free Software the proprietary world has a huge head start over us.
This is certainly a disaster for everyone involved.:(
Someone has got to stop the mechanics. They are FAR worse than computer people with their mumbo jumbo. I've spent tons of money on my car over the years and had O2 sensors replaced, manifolds cracked, bearings worn, compressors fail, clutches wear out, framistats out of alignment, interociters fail, and who knows what else. WTF is all of this stuff? Why should I have to know all of the technical stuff just to drive a car? And they LOVE to use these big words to demonstrate their superior mechanical ability and make us mere mortals feel stupid. Lots more people have cars than computers and we all have to take them in for one thing or another every few months. Well I for one have HAD IT! Next time my mechanic uses these big words on me I am going to insist he speak in normal english or I will refuse to pay! If he can't explain it he must be cheating me.
This management style only works if you have very good self-motivated developers. With such high standards and ability to attract good people I can see how they can pull it off. But don't think this will work in your average IT shop.
You have written a program. It does what you need it to do and helps make your life easier. That program could make my life easier also. I ask for a copy. You could easily give me one right then and there and it wouldn't cost you but a few electrons. You say no. You are now an unhelpful asshole.
Clear enough?
Richard Stallman has clearly explained this many times in many different ways over many years. Surely one of them resonates with you.
And FWIW I don't think people who write commercial software are evil, they just have neutral karma. Unless they are spammers or writing DRM or viruses or some other form of code that actively hurts people. But I also don't think they are as good as people who write Free software. They are saints in my book.
Because the DoD allegedly likes freedom and wants to promote it. It is their reason for existance. If "Open Source" is hurting the adoption effort use the original name "Free Software".
I thought they cost $100 in materials so paying $200 was just a for example but apparently I am not generous enough for you. Why don't you pay $10,000 so 99 poor kids can have a laptop? Don't be so stingy!
Laptops tend to have a lot of laptop specific hardware. And it seems like testing on the target platform would be a good idea. These things have pretty minimal cpu's. I can't even be sure my app will run at a reasonable speed. They aren't even x86 are they? In fact since we can't get ahold of one we don't know what software is on them so we do not know what dependencies we can count on. What a mess!
The OLPC website says they will only be available to schools and governments. How will anyone ever develop software for it? Why can't I pay $200 for one and have $100 of that go towards subsidizing a laptop for some other kid?
Our disks are SATA hot swap disks in a 4 bay hot swap 1u case. So when a single disk fails I just pull it out and put in another one. No need to bring down the whole set of 4 disks. Linux's RAID support is pretty sweet.:)
How about configuring the Clariion disk arrays? We had to have a guy from EMC in for a whole day to do that. How about installing the fibrechannel HBA's and drivers? That was fun. How about configuring the fibrechannel switch? You have to learn how to fence off crashed or misbehaving hosts etc. Much easier to do with a switch where I already know how to turn off the switch port. I could go on... Ethernet expertise are far more plentiful than fibrechannel expertise.
I think you are probably looking at the cost to buy Coraid's gear. You do not have to buy their stuff, although I am sure that they prefer that you do. I built my own AoE SAN using regular PC's. Way cheaper. I take the google approach: Use a larger amount of commodity hardware and design the system in an intelligent way to achieve the same performance and reliability at a better price/performance. Coraid hardware is basically just a Linux box with disks exporting AoE volumes. The nice thing about it is that you get their support. But AoE is so simple that you generally don't need support beyond perhaps the mailing list.
1) Complexity for RAID and volume management is not centralized and is pushed to individual hosts. One of the main benefits of SAN technology is that you can just carve out storage from a single interface and assign it to a server and the server simply sees it as a block device. With AoE each drive is addressed separately by the server, which means it is up to the server (and server admin) to figure out how to handle distributing over multiple drives, handle drive failures, and expanding volumes. This is huge.
AoE is a SAN technology and it is no different from other SAN technologies in this regard. AoE just provides a block device. Just like Fibrechannel or iSCSI or any other SAN technology. If you want RAID and volume management centralized instead of being pushed to individual hosts do what the Fibrechannel guys (and everyone else) does: Attach all of the disk to one host and use that host to aggregate it and manage it and export it to the rest of your clients. Even in the big Clariion disk systems from EMC and other SAN products they contain an internal PC which aggregates all of the disk for presentation to other hosts on the SAN.
2) It is not a standard and is only really supported by one vendor. This may change in the future but it is significant right now. It is registered with the IEEE but that hardly makes it a peer-reviewed standard with input/improvements from many experts.
It is a spec released to the public without strings attached. GPL'd implementations of both target and initiators exist. Lots of people have looked it over to the extent that it has even been included in the stable Linux kernel.
3) No boot from SAN. Until someone makes some sort of mini bootstrap system on a CD or a hardware card implementation of AoE that can be addressed as a block device admins will be unable to host the root filesystem and/or C: drive on an AoE SAN
Uh...I have a number of machines in a rack right across the hall from me which do indeed boot from SAN. All you need to do is tftp an initrd containing the AoE driver and boot with that.
4) No multipath (that I can see). Perhaps I misunderstand this, but it seems like there is no way to do multipath IO with this system. That is, all the drives are single-connected to a network. If that network switch goes down, all drives on that network are inaccessible.
How is this any different from fibrechannel? IIRC fibrechannel disks have a single connection also. What most systems do is mirror or RAID them. What I have done is I have made a mirrored pair of AoE disks on different switches and use Linux's md driver to mirror them.
So AoE looks like a neat technology for pushing drives out of the box and potentially sharing them among hosts, but there is no intelligence there. It is just dumb block addressable storage with no added availability or management, and therefore is far from being an iSCSI or FC killer.
Fibrechannel or iSCSI by themselves are just dumb block addressable storage also. The management layer generally has little to do with the technology that implements the block layer.
I don't know if the RFC has been ratified but the AoE spec if definitely freely available from coraid's site. It is 8 pages long compared to iSCSI's 257 which I really like. There are open source implementations of AoE targets and initiators. It is not the domain of one company and can never be. As far as I am concerned it is already a serious contender.
AoE rocks. It is very easy to set up, way simpler than iSCSI or fibrechannel or any other SAN technology I have used. And it enabled us to have many more options for high availability or clustered filesystems (which we are not yet using but I have been following the progress of GFS and Lustre, learning towards Lustre). We did not buy the Coraid stuff but instead used vblade on our own disk machines. A disk node in our cluster has 4 300G SATA disks which we RAID 5, 512M RAM, and the cheapest CPU Intel currently makes. We have dual core Opterons with 4G of RAM each with no internal disk. They PXE boot and then mount root straight off the AoE. Then we run Xen on the Opteron boxes. This is the killer setup. We can migrate xen domains avoiding downtime for hardware maintenance and if a machines dies we can instantly restart it on another machine because it all runs off the AoE SAN.
So far I am very pleased. Just make sure you get hardware that can do jumbo frames as this will increase your performance by 50%.
Slashdot Mirror
...is the only way we are going to take advantage of multi-core cpu's and continue to improve our software. Only through purely functional code can you make guarantees about what can be executed simultaneously and let the machine sort it all out. I'm learning Haskell for this very reason.
I spent 9 months in HCMC as the director of software development for a US based company last year. It wasn't my idea. The Vietnamese-American owner of the company who employed me to go thought it was the thing to do. I can tell you that it is NOT the place to outsource any IT tasks such as programming or web design to. Not unless you have some serious government contacts to get you access to the smartest kids out of the state schools (who will still have only minimal programming knowledge and only on Windows).
I couldn't find anyone there who spoke decent english who knew anything about computers. The best I could find were straight out of two year trade school/junior college amature windows jockeys. Linux? Perl? Fat chance! It is still very much a third world country. Software is pirated wildly too. Don't expect employees to obey any sort of NDA. Also note that since people there do not have credit cards, car payments, mortgages, and are already heavily dependent on their families for most things they need they are usually free to leave your company at any time.
At least they have cable modem in HCMC even though it can be a bit unreliable. Exepect a power outage once a month too. Expect theft. I have had motorbikes stolen, cell phones stolen, etc.
And the corruption...oh my god. We paid off everyone and were solicited for payoffs by everyone. My coworker overstayed his visa by a day. They wouldn't let him out of the country! The soldier/immigrations officer/policeman (all the same there) took him into a side room and basically asked him how much money he had on him. $60 worth of the local currency (Vietnamese Dong) and he was free to go. We paid $400 in cash to the customs guy to get them to let our IP phones into the country when the official tax on them was supposed to have been much higher.
And on top of it all they are still very much communist and most are quite brainwashed. It is in a similar vein to North Korea only not as extreme. Americans are lazy people who cheat on their wives and fuck in the streets and cannot be trusted. They do not know about nuclear weapons, don't know the cold war, don't really know anything about the world context in which the Vietnam War happened. Everyone treated me very nicely of course. No anti-Americanism at all as long as someone stood to gain money from me and I was paying in cash. They are always very friendly to tourists and smiling and respectful. Just don't try to date anyone there or talk politics with anyone as you will surely offend. If you hear someone say something about history which you know is patently false just smile and nod.
Suffice it to say the project did not go well. Doomed from the start. At least I had the good sense to bail months before the shit really hit the fan and the whole operation collapsed.
Vietnam is a fun place to visit and I recommend it. I will be going back there again in a couple of weeks for Christmas. I just won't be doing business there again until the business culture changes dramatically.
Carbon Dioxide + Water + Light energy Glucose + Oxygen + Water
Isn't there a way we can do this on a massive scale and remove carbon dioxide from the atmosphere? And I don't mean plant forests because that isn't likely to happen due to the space they require. Synthesize our own chlorophyll and do it much more efficiently than plants can? Or perhaps skip chlorophyll altogether and go with some other means of using light (or even use nuclear power if some other energy source would be more efficient) to implement the carbon reduction reaction in an even simpler way than chlorophyll does it which can allow us to sequester the carbon.
Or perhaps it would be simpler to find some way to store carbon dioxide. But because it is a gas that sounds troublesome.
Looks like the guys at MIT and various people in industry are looking into the problem: http://sequestration.mit.edu/
Ultimately humanity is going to have to take active control over the climate of our planet. Nothing can be left to nature anymore because as part of nature we would destroy ourselves.
The original poster was correct. It is true that MAC addresses are known not to be unique. They SHOULD be unique but they are not. I once received a dozen motherboards all with the same MAC. Intel server-class motherboards. Not prototypes or anything. The board had been in production for nearly a year. They were set up one at a time and deployed into a huge cluster in the datacenter and forgotten about. Tracking down the mac/ip collisions over the next year as machines with the same MAC addresses happened to find themselves on the same network was not fun. So although we can say that MAC's should be the same we cannot unequivocally state that they are.
This is gross abuse of the word "trivial".
I'm not saying it is unmaintainable just that nobody may choose to maintain it. Also the mailing list may actually be working. It seems my MTA was picking the lower priority MX which apparently isn't properly configured to accept mail for that domain or something. And finally, as for my company we will probably just migrate off of reiserfs during our upcoming move to a Xen based architecture/server consolidation.
The doom and gloom is that reiserfs was a project coded mostly by paid coders in Russia and Hans himself with few other contributors. It is open source in license but not much else.
The other problem is the negative publicity this gets reiserfs. After this I will never again be able to mention reiserfs without hearing comments about murderers. Whether he did it or not won't matter. Every time reiserfs comes up this incident will be mentioned.
Not just anyone can take over the project. Reiserfs is a very complicated codebase and he made some controversial design decisions. It has had a hard time getting into the kernel.
Knowledge in someones head is extremely difficult to replace. Linus would be more easily replaceable in the context of the Linux kernel than Hans in the context of reiserfs because there are a lot more independent people contributing to the kernel who understand it. Not many outside the project contribute to reiserfs. The vast majority of the work seems to come from the paid Russian coders. Hopefully SuSe will step in and support it instead of dump it. It seems like they will have to do either one or the other. Time will tell...
Oops...just ran across this:
http://linux.wordpress.com/2006/09/27/suse-102-di
It seems SuSe isn't too happy with reiserfs technology anymore and I'm sure this event won't help. So we can probably count them out. Looks like Linspire is their next biggest sponsor. Think they will step up to the plate?
There has been a discussion on the reiserfs mailing list about this also but either I was just banned from it for talking about the code instead of talking about the kids (isn't political correctness a drag?) or they have shut down the list:
: host mx.comstar.ru[83.242.139.27] said: 554
: Client host
rejected: Access denied (in reply to RCPT TO command)
As a person who helped Hans Reiser get some sponsorship in his early days and an early adopter of his filesystem in major corporate use I never would have expected something like this. It's a disaster for him but there isn't much we can do about it at this point aside from debate his innocence based on zero information. So what about Linux? Even if he turns out to be innocent (and I hope he is) the name is tarnished and the filesystem will probably languish. I was expecting reiserfs4 to be an important part of the future of Linux and Free Software's answer to WinFS. Now what will we do? We all know it takes ages, years even, to design, implement, and test a filesystem. XFS, JFS, ext3, etc. are nothing like reiser4 and lack it's capabilities. WinFS will someday be ready and will someday ship. And with this setback for Free Software the proprietary world has a huge head start over us.
:(
This is certainly a disaster for everyone involved.
My post was a joke to make a point. You seem to have missed the humor.
Someone has got to stop the mechanics. They are FAR worse than computer people with their mumbo jumbo. I've spent tons of money on my car over the years and had O2 sensors replaced, manifolds cracked, bearings worn, compressors fail, clutches wear out, framistats out of alignment, interociters fail, and who knows what else. WTF is all of this stuff? Why should I have to know all of the technical stuff just to drive a car? And they LOVE to use these big words to demonstrate their superior mechanical ability and make us mere mortals feel stupid. Lots more people have cars than computers and we all have to take them in for one thing or another every few months. Well I for one have HAD IT! Next time my mechanic uses these big words on me I am going to insist he speak in normal english or I will refuse to pay! If he can't explain it he must be cheating me.
This management style only works if you have very good self-motivated developers. With such high standards and ability to attract good people I can see how they can pull it off. But don't think this will work in your average IT shop.
You have written a program. It does what you need it to do and helps make your life easier. That program could make my life easier also. I ask for a copy. You could easily give me one right then and there and it wouldn't cost you but a few electrons. You say no. You are now an unhelpful asshole.
Clear enough?
Richard Stallman has clearly explained this many times in many different ways over many years. Surely one of them resonates with you.
And FWIW I don't think people who write commercial software are evil, they just have neutral karma. Unless they are spammers or writing DRM or viruses or some other form of code that actively hurts people. But I also don't think they are as good as people who write Free software. They are saints in my book.
Because the DoD allegedly likes freedom and wants to promote it. It is their reason for existance. If "Open Source" is hurting the adoption effort use the original name "Free Software".
Evil Spock has my pr0n collection!!!!! :(
Actually it's called Beehive/Cobweb: http://www.cs.cornell.edu/People/egs/beehive/
I thought they cost $100 in materials so paying $200 was just a for example but apparently I am not generous enough for you. Why don't you pay $10,000 so 99 poor kids can have a laptop? Don't be so stingy!
Laptops tend to have a lot of laptop specific hardware. And it seems like testing on the target platform would be a good idea. These things have pretty minimal cpu's. I can't even be sure my app will run at a reasonable speed. They aren't even x86 are they? In fact since we can't get ahold of one we don't know what software is on them so we do not know what dependencies we can count on. What a mess!
They say clearly on their website that these things will not be for sale to individuals. Only governments and schools. So that is not the reason.
The OLPC website says they will only be available to schools and governments. How will anyone ever develop software for it? Why can't I pay $200 for one and have $100 of that go towards subsidizing a laptop for some other kid?
Our disks are SATA hot swap disks in a 4 bay hot swap 1u case. So when a single disk fails I just pull it out and put in another one. No need to bring down the whole set of 4 disks. Linux's RAID support is pretty sweet. :)
How about configuring the Clariion disk arrays? We had to have a guy from EMC in for a whole day to do that. How about installing the fibrechannel HBA's and drivers? That was fun. How about configuring the fibrechannel switch? You have to learn how to fence off crashed or misbehaving hosts etc. Much easier to do with a switch where I already know how to turn off the switch port. I could go on... Ethernet expertise are far more plentiful than fibrechannel expertise.
I think you are probably looking at the cost to buy Coraid's gear. You do not have to buy their stuff, although I am sure that they prefer that you do. I built my own AoE SAN using regular PC's. Way cheaper. I take the google approach: Use a larger amount of commodity hardware and design the system in an intelligent way to achieve the same performance and reliability at a better price/performance. Coraid hardware is basically just a Linux box with disks exporting AoE volumes. The nice thing about it is that you get their support. But AoE is so simple that you generally don't need support beyond perhaps the mailing list.
AoE is a SAN technology and it is no different from other SAN technologies in this regard. AoE just provides a block device. Just like Fibrechannel or iSCSI or any other SAN technology. If you want RAID and volume management centralized instead of being pushed to individual hosts do what the Fibrechannel guys (and everyone else) does: Attach all of the disk to one host and use that host to aggregate it and manage it and export it to the rest of your clients. Even in the big Clariion disk systems from EMC and other SAN products they contain an internal PC which aggregates all of the disk for presentation to other hosts on the SAN.
It is a spec released to the public without strings attached. GPL'd implementations of both target and initiators exist. Lots of people have looked it over to the extent that it has even been included in the stable Linux kernel.
Uh...I have a number of machines in a rack right across the hall from me which do indeed boot from SAN. All you need to do is tftp an initrd containing the AoE driver and boot with that.
How is this any different from fibrechannel? IIRC fibrechannel disks have a single connection also. What most systems do is mirror or RAID them. What I have done is I have made a mirrored pair of AoE disks on different switches and use Linux's md driver to mirror them.
Fibrechannel or iSCSI by themselves are just dumb block addressable storage also. The management layer generally has little to do with the technology that implements the block layer.
I don't know if the RFC has been ratified but the AoE spec if definitely freely available from coraid's site. It is 8 pages long compared to iSCSI's 257 which I really like. There are open source implementations of AoE targets and initiators. It is not the domain of one company and can never be. As far as I am concerned it is already a serious contender.
AoE rocks. It is very easy to set up, way simpler than iSCSI or fibrechannel or any other SAN technology I have used. And it enabled us to have many more options for high availability or clustered filesystems (which we are not yet using but I have been following the progress of GFS and Lustre, learning towards Lustre). We did not buy the Coraid stuff but instead used vblade on our own disk machines. A disk node in our cluster has 4 300G SATA disks which we RAID 5, 512M RAM, and the cheapest CPU Intel currently makes. We have dual core Opterons with 4G of RAM each with no internal disk. They PXE boot and then mount root straight off the AoE. Then we run Xen on the Opteron boxes. This is the killer setup. We can migrate xen domains avoiding downtime for hardware maintenance and if a machines dies we can instantly restart it on another machine because it all runs off the AoE SAN.
So far I am very pleased. Just make sure you get hardware that can do jumbo frames as this will increase your performance by 50%.