I didn't guess it would be cheaper, I did the math. By using a FC san to back our vmware cluster, we're seeing as many as 15 guests on dual socket X5550 equipped machines with 32-64GB of RAM (depending). Trust me, the SAN pays for itself.
Uhhhhhh, $30/GB fiber channel SAN is not for your fileserver. It's for your massive high volume transactional databases, your VMFS for your vmware clusters, etc etc.
All the crap you're talking about should be on $3/GB NetApp iSCSI SAN full of SATA disks.
My math is simple, if SATA is 2-3x slower than fiber channel, then to deliver the same IOPS you need 2-3x more sata disks. I'm not sure what you're not understanding. Some (all) of us need to consider performance, not just total disk space. This becomes not only GB/watt, GB/sq ft, but also IOPS/sq ft and eventually $/sq ft to deliver storage AND IOPS.
Take a look at 3PAR sometime. No short stroking, they take all of your "virtual LUNs", divide them into 256MB "chunklets" and spread them across the entire SAN. So, no hot spots, and you can increase the performance of every LUN by adding more disks and running Dyanmic Optimization, which spreads all those chunklets across all your new disks. I guess there's a good reason why 3par is #1 on the SPC benchmarks.
You clearly don't understand the cost per square foot of datacenter space, in terms of lease/ownership, cooling and power. I think I'll stick with 5 racks of fiber channel san(s) vs 15 racks of SATA disks failing all day.
I have never, ever seen a fiber channel san that wasn't running some kind of multi-level RAID (eg - RAID10, RAID50 or RAID60). The only case I can think of where you'd spend all that money on fiber channel but put it in RAID5 would be on some OLAP workload that was 99% reads. (Obviously because of the RAID5 write penalty)
You really can't compare the cost per gigabyte of individual disks vs the cost per gigabyte of a SAN, it's just not that simple, you have to look at the entire solution. SATA storage space is around $0.10 per gigabyte, but a very fast high end SAN (fiber channel, redundant controllers, FC switches, HBA's, etc) will run you easily $30/GB.
But when I can consolidate an entire datacenter into a single rack with a blade enclosure, vmware and a 3par SAN, all the sudden the entire solution becomes dramatically cheaper, easier to manage, easier to cool, etc, etc, etc.
We have a name for that, it's called "online" and "nearline" storage. What I think interesting now is that some SAN vendors are using SSD's and calling it a "tier 0" and sticking it between the controller cache and the online storage.
It's really not that simple. I can deliver 1,000,000 IOPS with an HP DL785 and 6 ioDrives. It would take approximately 4,000 2.5" 15K RPM SAS drives to deliver the same number of IOPS. And that's in RAID0. In RAID10 it would take 8,000 drives. Do you have any idea how much 8,000 drives would cost? Let's go with one the smallest commercially available enterprise 2.5" SAS drive, HP 36GB 2.5" SAS at $350. Even in RAID0 with 4,000 drives you're talking $1.4 million JUST IN DRIVES! In RAID10 we're talking about $2.8 million. You can get 6 640GB ioDrive DUO's for $60,000 and put them in a single $30,000 server, and deliver those IOPS out of the PCI-e BUS (32GB/s IIRC, thats a big B, as in bytes).
Now let's talk enclosures, how about an HP MSA50 with 10 drives per U. Now we need 400U, or 9 and a half racks worth of disks. Now we need controllers, cabling, power and cooling.
Now, obviously we're talking 1-2TB worth of ioDrive storage vs 144TB worth of spinning disk storage. Of course we're also talking about $4 million vs $100k, even before power/cooling.
Point being, workload is a real consideration here.
In the 18 months since we Implemented Ironport C150's I've gotten ZERO false positives and one false negative (actual spam). We have about 30mb/s of Internet bandwidth across two carriers which averages less than 1mb/s of utilization.
If there's a spam war, I'm completely and totally unaware of it, as are all of our users. Granted, we filter about a quarter of a million spam messages and let through about 2500 legitimate emails a day, but the impact on our resources is negligible. We actually needed the C150's just to meet regulatory requirements (email encryption), spam filtering was just a freebie. We dropped our other spam filtering software and never looked back.
"hand guns require the waiting period and a permit"
No permit in Florida, unless you choose to get a Carry Concealed Weapons (CCW) Permit. And actually, with a CCW you get to bypass the waiting period on handguns.
I'm not sure where you found them for $90 but MSRP is closer to $100-$110. There's really no point in comparing anything other than MSRP or this would go on forever, trading ebay links back and forth.
His concerns are IOPS and latency. For my money, I'll take two OCZ MLC 60GB SSD's in RAID0 for $300 for a total of 120GB, just as he requires. It would provide vastly better performance for his workload. Of course, as he said, price isn't even a concern so really how about 4x64GB Intel SLC SSD in RAID10 for $3,396. I'd love to see how many spinning disks it would take to provide the same IOPS - obviously you won't ever achieve the same latency with spinning magnetic disks.
I'd also backup my work frequently like I would with any disk setup. You're much more likely to have a "human" error (accidentally deleting a file) than you will a hardware failure in either case. No RAID can save you from: $rm -rf . /
Two Intel X25-E SLC can pump out more than the SATAII bus can handle and it can do it for ~$840 using a simple onboard raid controller. Your solution to saturate the SATA bus (which a _fraction_ of the non-sequential transfer speed) requires:
- 4 Hard Disk Drives
- 1 External enclosure
- RAID controller that supports RAID10
- Power and noise
And still can't deliver the same random read performance of a pair of of SSD's.
RAM is not cheaper than SSD why the fuck do people keep saying this? Can you not do simple math? The cheapest 2GB DDR2-800 DIMM on newegg is $17.99. Now let's forget the fact that you can only get, at most, about 8 of these things in a PC for a total of 16GB (which includes system memory). Trust me, you don't want to go to 4GB or 8GB DIMMs, it will just make you look even worse.
That would cost you: $143.92
Remaining expansion? None.
Here's a 16GB MLC Super Talent for $74.99 (HALF THE PRICE OF THE RAM):
That's a horrible, HORRIBLE idea. Hardware is cheap and developers are expensive, slowing them down is like shoveling money into a fire place. You just need to be a cheap, shitty test environment that they can run their compiled code on.
At $3,000 for 80GB you're referring to Fusion's ioDrive. We're talking about something like an Intel X25-E SLC drive. The difference is in the host bus interface - the Fusion ioDrive uses the PCI-e bus for transfer whereas normal NAND flash-based SSD's (like the Intel) use the SATA[II] interface and are _DRAMATICALLY_ cheaper.
Just bored and had a random thought for super cheap super easy "manual HA" (oxymoron?). Round robin DNS is usually a hassle because of TTL values. So let's setup something using round robin DNS, no need for front end load balancer where each node essentially acts as a peer. Basically if you have a few nodes today using round robin DNS you can implement this for free, today. The advantage is you won't have to wait for TTL values to be flushed from DNS caches, so when you have a problem you can remove it immediately.
Assuming linux/bsd/etc:
1. create ifcfg-bond0:1-n on every host, each of which represents a node in the cluster. Now each node can start a sub-interface and answer requests for any other node. Also create one specific iface for management on each node.
2. Setup round-robin DNS: add IP address of each node to the A record for www.
3. If one node is down, or needs to go down, just start the appropriate ifcfg-bond0:X iface on another node and if necessary shut that iface down on the node that needs work. Use the management address you created above to perform any maintenance and get it back online.
Of course this could all be automated pretty easily using a heartbeat. Each node tests the services on other nodes, nodes all agree when a specific node has died and determine the appropriate host to take over the load.
Slashdot Mirror
Numerous benefits: http://www.storusint.com/pdf/SCSI_vs_FC.pdf
More bandwidth, more nodes on the link, blah blah.
I didn't guess it would be cheaper, I did the math. By using a FC san to back our vmware cluster, we're seeing as many as 15 guests on dual socket X5550 equipped machines with 32-64GB of RAM (depending). Trust me, the SAN pays for itself.
Uhhhhhh, $30/GB fiber channel SAN is not for your fileserver. It's for your massive high volume transactional databases, your VMFS for your vmware clusters, etc etc.
All the crap you're talking about should be on $3/GB NetApp iSCSI SAN full of SATA disks.
My math is simple, if SATA is 2-3x slower than fiber channel, then to deliver the same IOPS you need 2-3x more sata disks. I'm not sure what you're not understanding. Some (all) of us need to consider performance, not just total disk space. This becomes not only GB/watt, GB/sq ft, but also IOPS/sq ft and eventually $/sq ft to deliver storage AND IOPS.
Take a look at 3PAR sometime. No short stroking, they take all of your "virtual LUNs", divide them into 256MB "chunklets" and spread them across the entire SAN. So, no hot spots, and you can increase the performance of every LUN by adding more disks and running Dyanmic Optimization, which spreads all those chunklets across all your new disks. I guess there's a good reason why 3par is #1 on the SPC benchmarks.
And compressed LTO4 is 1.6TB and you can get a tape for $50 or $60.
You clearly don't understand the cost per square foot of datacenter space, in terms of lease/ownership, cooling and power. I think I'll stick with 5 racks of fiber channel san(s) vs 15 racks of SATA disks failing all day.
I have never, ever seen a fiber channel san that wasn't running some kind of multi-level RAID (eg - RAID10, RAID50 or RAID60). The only case I can think of where you'd spend all that money on fiber channel but put it in RAID5 would be on some OLAP workload that was 99% reads. (Obviously because of the RAID5 write penalty)
You really can't compare the cost per gigabyte of individual disks vs the cost per gigabyte of a SAN, it's just not that simple, you have to look at the entire solution. SATA storage space is around $0.10 per gigabyte, but a very fast high end SAN (fiber channel, redundant controllers, FC switches, HBA's, etc) will run you easily $30/GB.
But when I can consolidate an entire datacenter into a single rack with a blade enclosure, vmware and a 3par SAN, all the sudden the entire solution becomes dramatically cheaper, easier to manage, easier to cool, etc, etc, etc.
We have a name for that, it's called "online" and "nearline" storage. What I think interesting now is that some SAN vendors are using SSD's and calling it a "tier 0" and sticking it between the controller cache and the online storage.
"$-for-$ there is no contest. HDDs win."
It's really not that simple. I can deliver 1,000,000 IOPS with an HP DL785 and 6 ioDrives. It would take approximately 4,000 2.5" 15K RPM SAS drives to deliver the same number of IOPS. And that's in RAID0. In RAID10 it would take 8,000 drives. Do you have any idea how much 8,000 drives would cost? Let's go with one the smallest commercially available enterprise 2.5" SAS drive, HP 36GB 2.5" SAS at $350. Even in RAID0 with 4,000 drives you're talking $1.4 million JUST IN DRIVES! In RAID10 we're talking about $2.8 million. You can get 6 640GB ioDrive DUO's for $60,000 and put them in a single $30,000 server, and deliver those IOPS out of the PCI-e BUS (32GB/s IIRC, thats a big B, as in bytes).
Now let's talk enclosures, how about an HP MSA50 with 10 drives per U. Now we need 400U, or 9 and a half racks worth of disks. Now we need controllers, cabling, power and cooling.
Now, obviously we're talking 1-2TB worth of ioDrive storage vs 144TB worth of spinning disk storage. Of course we're also talking about $4 million vs $100k, even before power/cooling.
Point being, workload is a real consideration here.
In the 18 months since we Implemented Ironport C150's I've gotten ZERO false positives and one false negative (actual spam). We have about 30mb/s of Internet bandwidth across two carriers which averages less than 1mb/s of utilization.
If there's a spam war, I'm completely and totally unaware of it, as are all of our users. Granted, we filter about a quarter of a million spam messages and let through about 2500 legitimate emails a day, but the impact on our resources is negligible. We actually needed the C150's just to meet regulatory requirements (email encryption), spam filtering was just a freebie. We dropped our other spam filtering software and never looked back.
"there are still some huge advantages to x64 in terms of memory management."
Really?
It's not too late to change your story.
"hand guns require the waiting period and a permit"
No permit in Florida, unless you choose to get a Carry Concealed Weapons (CCW) Permit. And actually, with a CCW you get to bypass the waiting period on handguns.
Kind of moot when you smoke it using a water filtered pipe (read: bong) or a vaporizer. Or just plain old eating it.
(disclaimer - I don't smoke pot, I'm all grown up now)
I'm not sure where you found them for $90 but MSRP is closer to $100-$110. There's really no point in comparing anything other than MSRP or this would go on forever, trading ebay links back and forth.
His concerns are IOPS and latency. For my money, I'll take two OCZ MLC 60GB SSD's in RAID0 for $300 for a total of 120GB, just as he requires. It would provide vastly better performance for his workload. Of course, as he said, price isn't even a concern so really how about 4x64GB Intel SLC SSD in RAID10 for $3,396. I'd love to see how many spinning disks it would take to provide the same IOPS - obviously you won't ever achieve the same latency with spinning magnetic disks.
I'd also backup my work frequently like I would with any disk setup. You're much more likely to have a "human" error (accidentally deleting a file) than you will a hardware failure in either case. No RAID can save you from: $rm -rf . /
Depends on the workload. When you try and compare nothing other than total throughput it doesn't look as impressive. Try comparing IOPS sometime :)
Two Intel X25-E SLC can pump out more than the SATAII bus can handle and it can do it for ~$840 using a simple onboard raid controller. Your solution to saturate the SATA bus (which a _fraction_ of the non-sequential transfer speed) requires:
- 4 Hard Disk Drives
- 1 External enclosure
- RAID controller that supports RAID10
- Power and noise
And still can't deliver the same random read performance of a pair of of SSD's.
"Although they use an SSD for another purpose, they said currently SSD's last about 6 months under heavy read/write conditions"
Link please.
RAM is not cheaper than SSD why the fuck do people keep saying this? Can you not do simple math? The cheapest 2GB DDR2-800 DIMM on newegg is $17.99. Now let's forget the fact that you can only get, at most, about 8 of these things in a PC for a total of 16GB (which includes system memory). Trust me, you don't want to go to 4GB or 8GB DIMMs, it will just make you look even worse.
That would cost you: $143.92
Remaining expansion? None.
Here's a 16GB MLC Super Talent for $74.99 (HALF THE PRICE OF THE RAM):
http://www.newegg.com/Product/Product.aspx?Item=N82E16820609330
Remaining expansion? LOTS
And it's our constant complaining and desire for improvement that drug us out of those dark ages by our bootstraps.
You're welcome.
That's a horrible, HORRIBLE idea. Hardware is cheap and developers are expensive, slowing them down is like shoveling money into a fire place. You just need to be a cheap, shitty test environment that they can run their compiled code on.
At $3,000 for 80GB you're referring to Fusion's ioDrive. We're talking about something like an Intel X25-E SLC drive. The difference is in the host bus interface - the Fusion ioDrive uses the PCI-e bus for transfer whereas normal NAND flash-based SSD's (like the Intel) use the SATA[II] interface and are _DRAMATICALLY_ cheaper.
Here's a 32GB Intel X25-E for $419.99: http://www.newegg.com/Product/Product.aspx?Item=N82E16820167013
$13.12/GB for the Intel SATA SSD
$37.50/GB for the Fusion ioDrive
Another thing: "those external enclosures generally aren't known for their performance."
That's very misleading - it depends on the interface. eSATA and SATA are exactly the same.
1) I can't get my TV shows in high def for free on the web without commercials
2) Watching movies while they're still in the theater.
Just bored and had a random thought for super cheap super easy "manual HA" (oxymoron?). Round robin DNS is usually a hassle because of TTL values. So let's setup something using round robin DNS, no need for front end load balancer where each node essentially acts as a peer. Basically if you have a few nodes today using round robin DNS you can implement this for free, today. The advantage is you won't have to wait for TTL values to be flushed from DNS caches, so when you have a problem you can remove it immediately.
Assuming linux/bsd/etc:
1. create ifcfg-bond0:1-n on every host, each of which represents a node in the cluster. Now each node can start a sub-interface and answer requests for any other node. Also create one specific iface for management on each node.
2. Setup round-robin DNS: add IP address of each node to the A record for www.
3. If one node is down, or needs to go down, just start the appropriate ifcfg-bond0:X iface on another node and if necessary shut that iface down on the node that needs work. Use the management address you created above to perform any maintenance and get it back online.
Of course this could all be automated pretty easily using a heartbeat. Each node tests the services on other nodes, nodes all agree when a specific node has died and determine the appropriate host to take over the load.