"FUD" is fear, uncertainty and doubt. By saying unproven, un-researched things like this you continue to promote a commonly held misconception about solid state drives. It's like two women at the hair salon gossiping, stop it. Either research it and come up with a useful contribution to the thread or just don't reply.
Exactly, in the context of SANs, you typically don't have a RAID controllers.
I think you underestimate performance requirements if you don't see a need for SSDs. Typical SANs operate in microsecond latencies across the cable plant, whereas mechanical disks have seek latencies in the milliseconds. Also SSDs throughput is already twice that of mecahnical disks and (read) IOPS aren't even comparable, SSDs are an order of magnitude faster. And just wait until we see fiber channel SSDs or SATA 6G SSDs that will be doing over 500MB/s right out of the gate.
Higher margins? Are you kidding me? The SAN vendors I've worked with, and I won't name names, just that their names rhyme with EMC and 3PAR, price 146GB FC drives at or over $1,000. And we're talking the exact same Hitachi drives you can buy for $500 anywhere else. But they're "certified" for the SAN and you have to buy them from the SAN vendor or there goes your warranty. It's absolutely criminal, pure highway robbery.
"The other funny thing you discover if you benchmark enough of these things is that a regular hard drive confined to only use as much space as a SSD provides is quite a bit faster too."
Yes there's an improvement, but to compare read IOPS from an enterprise SSD to a short-stroked SATA disk on a purely performance basis isn't even close. We're talking orders of magnitude slower.
I think SSDs really shine when you get into situations where your performance requirements vastly outweigh our capacity requirements. When you need 100k IOPS for 10TB (in a predominately read heavy workload) and you're short stroking 500 HDDs all the sudden a few shelves of SSDs start looking mighty attractive (failure rate, power, management -- cost/sqft).
"f a SSD craps out, which they WILL do (just look at reviews on newegg for proof of that) "
Intel offers a 3 year warranty on their new drives, most other vendors offer 1 (OCZ, Samsung, etc). I also know of at least 20 SSDs used by friends and family and I've yet to hear of one fail. So, unless you can provide some actual evidence - shut the fuck up.
"What's the cost-benefit analysis of buying hardware that has headroom for those.1% peak events, vs data housekeeping and app/sql profiling? This is a management problem, not a technical one."
Depends on the business requirements and the number of end-points, but I wouldn't rule it out completely. For example, production companies moving large amounts of video for short periods of time, it might be worth the difference between 4Gb and 8Gb fiber channel, I don't know. You're also assuming those peak events are only 1%, they could be 5%, 10% or even more. I'm not saying it's always warranted, just that it may be in some cases.
"How about testing them on web services like digg, or on company mail servers instead of fake throughput and "feel" tests?"
I've been waiting for the same thing, unfortunately SLC flashed-based drives (the more expensive NAND flash with the higher lifespan) is still exceptionally expensive. But, the good news is major SAN vendors are already offering SSD options. Everyone from EMC to Sun Microsystems is starting to include SSD drives in their storage products. While it would be very unusual for us to get a peek into the storage systems of companies like digg, etc, hopefully they'll filter down far enough that we can get some realistic reviews soon. I'm definitely looking forward to it.
Well I think (and the market agrees) we've got room for both models in the world. Although I do take some offense to the "All OSS fans really care about is destroying.." comment. I use commercial software and I believe it has a place in the market. But given the choice between competing products, I'll definitely take the open source (free) one:)
"All SSD's do is flip the Capacity/IOPS equation on the back end. Before, you ran out of drive IOPS before ran out of capacity. Now, you get to run out of capacity before you run out of IOPS on the drive side."
Thank you so much for summarizing that point so succinctly, I'm stealing that line, hope you don't mind:)
"Heck why not just build a raid 0 controller into the logic card with a sata connection and break the ssd into a bunch of little chunks and raid 0 them all"
Cost mostly, you'd need tons of controllers, cache, etc. Plus you can already nearly saturate SATA 3G with any decent SSD (Intel, Vertex, etc) so it's kind of pointless. The new Vertex and Intel SSDs are benchmarking at 250MB/s. Not point it making them much faster until we have SATA 6G.
You forgot about SSDs, consumer versions of which are already doing over 250MB/s reads for less than $3.00/GB. And we're still essentially talking about second generation products (Vertex switched from JMICRON to Indilinx controllers and Intel basically just shrunk down to 34nm for their new ones, although their old version did 250MB/s as well).
I'm using a 30GB OCZ Vertex for my main drive on my windows machine and it benchmarks around 230MB/s _AVERAGE_ read speed. It cost $130 ($4.30/GB) when I bought it a couple months ago, and prices are falling. The new Intel X25-M is $225 for 80GB ($2.81/GB).
Vertex (with Indilinx controllers) and Intel (even the "cheap" MLC drives from both vendors that are less than $3.00/GB) are seeing 250MB/s-270MB/s actual real world results for reads. The actual throughput of SATA 3G is actually slightly less than 300MB/s so essentially we're at the limitation of SATA 3G, or very very close -- too close for comfort.
Cost per IOPS yes, several vendors are selling SSD now. Cost per terrabyte, no, SSD isn't even close. What we're seeing now is a Tier 0 storage using SSD's. It fits in between RAM cache in SAN controller nodes and on-line storage (super fast, typically fiber channel storage vs near line).
So previously it looked like (slowest to fastest): SATA (near-line), Fiber Channel (online) -> RAM cache
Now we'll have: SATA -> FC -> SSD -> RAM
And in a few years after the technology gets better and much less expensive, we'll see: SATA -> SSD -> RAM
"Think your 4 GHz Intel box on a cheap DSL can outthink and outmanuever a couple networked Crays on the same T3 as the Exchange?"
Entirely possible, latency becomes the big problem here, not bandwidth. Cheap DSL in the same city as the exchanges could easily outperform an Internet T3 across the country. That'ss why some brokers provision massive point to point optical circuits (think SDH/SONET) to POPs near Exchanges (or directly into them?).
Good lord you pay $0.30/kWh? We've got to be talking commercial power in downtown NY right? I pay $0.05x/kWh for residential power. Most residential power, even in California I believe tops out around $0.15 - $0.20/kWh.
"While of course it's possible to produce a very neat cable layout when all cables are built to the exact length required, it's a lot faster, and generally cheaper, to use manufactured cables. But then you're left with the problem of what to do with various small coils of cable."
If we're talking intra-rack, I've built racks with both manufactured cables and custom lengthed cables and while custom lengthed is obviously much cleaner and nicer it's quite a bit of work to build at least 3x cable for each server (2x NIC + HP's iLO or w/e OOB management you're using). And if we're talking an average of 20 servers per rack that's at least 60 cables with connectors on both ends for a total of 120 crimps - ouch. We 1U use top-of-rack switches, usually 3, one for each NIC and one for iLO.
But, for manufactured cables, just find those vertical cable organizers that you can mount to rack itself, then just take your slack up there.
I'd seriously buying rolls of velcro and then cutting it to length with scissors. It's quick and easy, much cheaper, and the velcro length is always perfect. You can cut down 12" but dealing with one big roll is much easier.
We buy very large wooden spools (3 foot wide, the spool, not the tubing, tubing is about 2" in diameter) of this plastic corrugated tubing in bright orange for the datacenter. We open it lengthwise and attach it to the frame rails in the racks and use it to run fiber, works really well.
Slashdot Mirror
"FUD" is fear, uncertainty and doubt. By saying unproven, un-researched things like this you continue to promote a commonly held misconception about solid state drives. It's like two women at the hair salon gossiping, stop it. Either research it and come up with a useful contribution to the thread or just don't reply.
"[Network/GIGE/10GE/etc] -> SATA -> [SSD -> RAM]"
Exactly, in the context of SANs, you typically don't have a RAID controllers.
I think you underestimate performance requirements if you don't see a need for SSDs. Typical SANs operate in microsecond latencies across the cable plant, whereas mechanical disks have seek latencies in the milliseconds. Also SSDs throughput is already twice that of mecahnical disks and (read) IOPS aren't even comparable, SSDs are an order of magnitude faster. And just wait until we see fiber channel SSDs or SATA 6G SSDs that will be doing over 500MB/s right out of the gate.
Higher margins? Are you kidding me? The SAN vendors I've worked with, and I won't name names, just that their names rhyme with EMC and 3PAR, price 146GB FC drives at or over $1,000. And we're talking the exact same Hitachi drives you can buy for $500 anywhere else. But they're "certified" for the SAN and you have to buy them from the SAN vendor or there goes your warranty. It's absolutely criminal, pure highway robbery.
"The other funny thing you discover if you benchmark enough of these things is that a regular hard drive confined to only use as much space as a SSD provides is quite a bit faster too."
Yes there's an improvement, but to compare read IOPS from an enterprise SSD to a short-stroked SATA disk on a purely performance basis isn't even close. We're talking orders of magnitude slower.
I think SSDs really shine when you get into situations where your performance requirements vastly outweigh our capacity requirements. When you need 100k IOPS for 10TB (in a predominately read heavy workload) and you're short stroking 500 HDDs all the sudden a few shelves of SSDs start looking mighty attractive (failure rate, power, management -- cost/sqft).
"However, AFAIK SSDs aren't that reliable yet so the RAID controllers are still worth it."
Please stop spreading baseless FUD.
"f a SSD craps out, which they WILL do (just look at reviews on newegg for proof of that) "
Intel offers a 3 year warranty on their new drives, most other vendors offer 1 (OCZ, Samsung, etc). I also know of at least 20 SSDs used by friends and family and I've yet to hear of one fail. So, unless you can provide some actual evidence - shut the fuck up.
"What's the cost-benefit analysis of buying hardware that has headroom for those .1% peak events, vs data housekeeping and app/sql profiling? This is a management problem, not a technical one."
Depends on the business requirements and the number of end-points, but I wouldn't rule it out completely. For example, production companies moving large amounts of video for short periods of time, it might be worth the difference between 4Gb and 8Gb fiber channel, I don't know. You're also assuming those peak events are only 1%, they could be 5%, 10% or even more. I'm not saying it's always warranted, just that it may be in some cases.
"How about testing them on web services like digg, or on company mail servers instead of fake throughput and "feel" tests?"
I've been waiting for the same thing, unfortunately SLC flashed-based drives (the more expensive NAND flash with the higher lifespan) is still exceptionally expensive. But, the good news is major SAN vendors are already offering SSD options. Everyone from EMC to Sun Microsystems is starting to include SSD drives in their storage products. While it would be very unusual for us to get a peek into the storage systems of companies like digg, etc, hopefully they'll filter down far enough that we can get some realistic reviews soon. I'm definitely looking forward to it.
So you think your network is more secure than storing data on Google's servers. Interesting.
Well I think (and the market agrees) we've got room for both models in the world. Although I do take some offense to the "All OSS fans really care about is destroying.." comment. I use commercial software and I believe it has a place in the market. But given the choice between competing products, I'll definitely take the open source (free) one :)
Jesus christ freedom is fucking complicated.
"All SSD's do is flip the Capacity/IOPS equation on the back end. Before, you ran out of drive IOPS before ran out of capacity. Now, you get to run out of capacity before you run out of IOPS on the drive side."
:)
Thank you so much for summarizing that point so succinctly, I'm stealing that line, hope you don't mind
I'll take 4 drives in RAID10 please :)
"Heck why not just build a raid 0 controller into the logic card with a sata connection and break the ssd into a bunch of little chunks and raid 0 them all"
Cost mostly, you'd need tons of controllers, cache, etc. Plus you can already nearly saturate SATA 3G with any decent SSD (Intel, Vertex, etc) so it's kind of pointless. The new Vertex and Intel SSDs are benchmarking at 250MB/s. Not point it making them much faster until we have SATA 6G.
You forgot about SSDs, consumer versions of which are already doing over 250MB/s reads for less than $3.00/GB. And we're still essentially talking about second generation products (Vertex switched from JMICRON to Indilinx controllers and Intel basically just shrunk down to 34nm for their new ones, although their old version did 250MB/s as well).
I'm using a 30GB OCZ Vertex for my main drive on my windows machine and it benchmarks around 230MB/s _AVERAGE_ read speed. It cost $130 ($4.30/GB) when I bought it a couple months ago, and prices are falling. The new Intel X25-M is $225 for 80GB ($2.81/GB).
Vertex (with Indilinx controllers) and Intel (even the "cheap" MLC drives from both vendors that are less than $3.00/GB) are seeing 250MB/s-270MB/s actual real world results for reads. The actual throughput of SATA 3G is actually slightly less than 300MB/s so essentially we're at the limitation of SATA 3G, or very very close -- too close for comfort.
Cost per IOPS yes, several vendors are selling SSD now. Cost per terrabyte, no, SSD isn't even close. What we're seeing now is a Tier 0 storage using SSD's. It fits in between RAM cache in SAN controller nodes and on-line storage (super fast, typically fiber channel storage vs near line).
:)
So previously it looked like (slowest to fastest): SATA (near-line), Fiber Channel (online) -> RAM cache
Now we'll have: SATA -> FC -> SSD -> RAM
And in a few years after the technology gets better and much less expensive, we'll see: SATA -> SSD -> RAM
And hopefully eventually: SSD -> Memristors
"Think your 4 GHz Intel box on a cheap DSL can outthink and outmanuever a couple networked Crays on the same T3 as the Exchange?"
Entirely possible, latency becomes the big problem here, not bandwidth. Cheap DSL in the same city as the exchanges could easily outperform an Internet T3 across the country. That'ss why some brokers provision massive point to point optical circuits (think SDH/SONET) to POPs near Exchanges (or directly into them?).
Good lord you pay $0.30/kWh? We've got to be talking commercial power in downtown NY right? I pay $0.05x/kWh for residential power. Most residential power, even in California I believe tops out around $0.15 - $0.20/kWh.
"While of course it's possible to produce a very neat cable layout when all cables are built to the exact length required, it's a lot faster, and generally cheaper, to use manufactured cables. But then you're left with the problem of what to do with various small coils of cable."
If we're talking intra-rack, I've built racks with both manufactured cables and custom lengthed cables and while custom lengthed is obviously much cleaner and nicer it's quite a bit of work to build at least 3x cable for each server (2x NIC + HP's iLO or w/e OOB management you're using). And if we're talking an average of 20 servers per rack that's at least 60 cables with connectors on both ends for a total of 120 crimps - ouch. We 1U use top-of-rack switches, usually 3, one for each NIC and one for iLO.
But, for manufactured cables, just find those vertical cable organizers that you can mount to rack itself, then just take your slack up there.
I'd seriously buying rolls of velcro and then cutting it to length with scissors. It's quick and easy, much cheaper, and the velcro length is always perfect. You can cut down 12" but dealing with one big roll is much easier.
I pay less than $0.06/kWh I guess I'm really REALLY lucky.
Makes it a lot easier to calculate things using kWh instead of joules.
eg: 60w * 24 hours * 30 days = 43 kWh / $0.05 = $2.15/month for me to run a 60w lightbulb.
We buy very large wooden spools (3 foot wide, the spool, not the tubing, tubing is about 2" in diameter) of this plastic corrugated tubing in bright orange for the datacenter. We open it lengthwise and attach it to the frame rails in the racks and use it to run fiber, works really well.
Really stale bread is for bread crumbs or french onion soup!