eBay Deploys 100TB of SSDs, Cuts Rackspace By Half

Lucas123 writes "eBay's QA division was facing mounting performance issues related to its exponential growth of virtual servers, so instead of purchasing more 15k rpm Fibre Channel drives, the company began migrating over to a pure SSD environment. eBay said half of its 4,000 VMs are now attached to SSDs. The changeout has improved the time it takes the online site to deploy a VM from 45 minutes to 5 minutes and had a tremendous impact on its rack space requirements. 'One rack [of SSD storage] is equal to eight or nine racks of something else,' said Michael Craft, eBay's manager of QA Systems Administration."

Comments

depends if you are IO bound or need storage

[smash]

For sites like ebay i have no doubt this makes sense. For the average small business I suspect they are far less IO bound and need storage...

Re:depends if you are IO bound or need storage

[Anonymous Coward]

Any business dealing on the web (aka hosting) would benefit from ssd more due to the nature of hosting (lots of random reads).

That said, why not a hybrid enviroment with ssd on the forefront and hdd on the back. The right tool for the right job.

SSD for random light reads.
HHD for large sequential block reads

Re:depends if you are IO bound or need storage

[myurr]

Actually the vast majority of websites, even business websites, are really low traffic and they benefit far more from storage space (especially when shared with other sites) than speed. Operating system RAM caching will usually make up any performance deficit those kinds of low traffic sites may experience, where the majority of the traffic that does go to their sites tends to be read only and directed at only a few pages on any given day. Premature optimisation adds either (or both) complexity and expense, and is unnecessary for 90+% of the web.

Scalability is a nice problem to have, and the majority of websites would do an awful lot better if they worried about driving traffic more than they worried about scalability.

Re:depends if you are IO bound or need storage

[Anonymous Coward]

Pity SSD's thrash the pants of 15krpm drives in sequential reads as well as random.
a 15k fibre channel drive only puts out between 50 and 100mb/s depending on the place on disk
high performance ssd's will do read and write at 500mb/s through sata or sas

Re:depends if you are IO bound or need storage

[Anonymous Coward]

Yes but how many companies on the low end actually host their own servers? I'd imagine the majority go to a consultant or hosting company. In these cases (not that they have much say in the matter), a SSD would be better since it's shared (on the startup / low end) among many business owners.

If a company is hosting it's own server, it's because A) they require it or B) there it staff made a bad choice
In A) SSD is generally better
in B) Why are they even attempting it with all the extra issues of maintenance like power backup, software backup, hardware failures, etc

So the way I see it, the companies that DO have a choice, tends to be companies that would *generally* benefit from it.

Re:depends if you are IO bound or need storage

[TheRaven64]

How many small businesses have more than 1TB of data, in total? Unless you're in a business that does a lot of image or video work, you're probably generating less than 1GB per month. One drive - hard disk or solid state - will probably have a large enough storage capacity for a typical small business. The bigger problem is redundancy and backups, not total storage capacity.

Re:depends if you are IO bound or need storage

[hairyfeet]

I agree but there should be an up side to this, in that we'll get some solid data to see if Atwood at coding horror is correct that SSDs need to be judged on a hot/crazy scale due to the insanely high SSD failure rates.

The reason I personally won't recommend SSDs to customers or carry them myself is after having my two "must have teh benchmarkz!" gamer customer buy top o' the line SSDs both of them had the SSD fail without warning which for me is unacceptable. Sure they got them replaced under warranty, but so what? That didn't cover their downtime, the cost of the 1Tb drives they had to pick up to hold them over while they waited on the RMA, or the data they lost between their last backup and the SSD failure. in the end they went dual Raptors in RAID 0 with the 1TB as backup and game storage.

The nice thing about HDDs is in my experience one gets plenty of warning before they go tits up. The drive gets noisy, you get Windows delayed write failures, the drive starts running hot, you get SMART warnings, something. with both of the SSDs it was just...poof. Dead drive. on one I was able to retrieve a little bit of the data, on the other it wouldn't even show up in BIOS.

That is why unless I have a customer where IOPS is all like in TFA or someone that lives on a heavily mobile laptop AND has the time for daily backups or stores all their important data in the cloud to just stay away until they get the kinks worked out. it is still too new and IMHO they haven't really got the tech down yet, hence all the failures. I tell my customers to have a fat HDD with plenty of cache along with giving Windows 7 plenty of RAM (4Gb minimum, 8Gb is better) to really use Superfetch and preload their apps into memory and they'll be happy. Yeah it doesn't boot with SSD speed, but who boots anymore?

Re:depends if you are IO bound or need storage

[hawkinspeter]

I've seen plenty of traditional HDDs fail without warning. I'd be wary of looking for signs of impending failure on any kind of drive.

Remember, RAID is used to protect against drive failure. Dual Raptors in RAID0 is just asking for trouble - you're doubling the chance of failure. They'd be much better off with dual SSDs in RAID1 - much better performance and power consumption and better protection against sudden failure.

I'm worried about your customers that "don't have time" to perform backups. What do they do when their laptop is stolen? Maybe their work and thus their data doesn't have much value.

You might not think that SSDs are ready yet, but there are plenty of people/companies that know differently.

Re:depends if you are IO bound or need storage

[arglebargle_xiv]

Any business dealing on the web (aka hosting) would benefit from ssd

Because SSD is web scale.

Re:depends if you are IO bound or need storage

[vlm]

I'm worried about your customers that "don't have time" to perform backups. What do they do when their laptop is stolen? Maybe their work and thus their data doesn't have much value.

Or they do everything, including storage, online. I'll publicly admit I don't backup my work laptop... The only reason I have it is to SSH, and I'm not backing up gigs of stuff when all I basically need is "putty", which I already have on a flash drive, and is widely available on the internet.

Customs wants to search my laptop, OK, its basically a vanilla install with putty. And a lot of empty space.

Re:depends if you are IO bound or need storage

[DAldredge]

OCZ SSD drives?

Re:depends if you are IO bound or need storage

[hawkinspeter]

Actually, I tend to make sure that any work I do is either replicated elsewhere (checked into SVN, stored on DropBox, cloud storage etc) or is easy to re-create. HairyFeet's customer however, lost a load of work when their drive broke without warning.

I tend to think of hard drives as temporary storage - if the data is important, have it in multiple physical places.

Re:depends if you are IO bound or need storage

[somersault]

If one drive has enough storage capacity for all your data, then one drive has enough capacity for backups too.. I've just taken to using HDDs for backup now. No expensive tape drive necessary, and hard drives must easily be cheaper per GB by now too..

Re:depends if you are IO bound or need storage

[denis-The-menace]

In a workstation scenario, couldn't you RAID1 an SSD and a fast HD in case the SSD dies like hairyfeet's gamer customers?

Would the SSD be slowed down by the HD trying to keep up or would the RAID controller's Cache prevent that?

Re:depends if you are IO bound or need storage

[Anarke_Incarnate]

The article is not talking about shitty consumer level drives. Enterprise systems are built around resiliency and failure mitigation.

Re:depends if you are IO bound or need storage

[greed]

You're probably better off with a good software stack for that. You'd want to program it to always WRITE both, but READ from the SSD unless it gets an I/O error.

I have to support customers on RHEL 5 (and 4...) so I haven't kept up with all the latest in the Linux kernel MD driver. But I think the stuff is there in the kernel proper for it to do this, I'm not sure about a user interface to control it. If not, well, what's a few local patches among friends?

Re:depends if you are IO bound or need storage

[Guspaz]

And a 3.5" 15K SAS drive will do 120MB/s to 200 MB/s with an average of 170 MB/s, which is only very slightly slower than an SSD on a 3Gbps SAS or SATA2 connection will do. A very large 7200RPM drive comes pretty close too. Sequential read and write speeds is not a major factor in SSD performance advantages. That's not even a typical use case. It's when IOPS get involved that SSDs completely destroy HDDs.

Re:depends if you are IO bound or need storage

[petermgreen]

Write performnace would be limited by the slower mechanical drive since all writes must go to both drives.

Read performance would depend entirely on how well the RAID distributed reads between the drives.

Re:depends if you are IO bound or need storage

[bobaferret]

Having finished an upgrade to SSD's for our website and esp or DB servers several months ago. One of the things I found that has really made a difference, is in the ability of RAID controllers to use and understand a mix of SSD and HDD drives. LSI makes a nice controller that will move frequently used files to the SSD drives. We didn't go this route however. We found that we could replace 14 raid 10 HDD drives with 4 raid 10 SDD drives. The power saving are amazing, the heat differences are great, and the speed is __amazing__. Our queries run about 1000 times faster. We've always had too much active data to just be able to throw pure memory at it. It's also been a balanced mix of random and sequential access needs. SSDs fit our needs perfectly. Now we're just going to have to see what the reliability is like. So far so good.... We're also fairly high traffic at 5.6 queries/sec avg. and about 8.6 inserts/sec avg.

Re:depends if you are IO bound or need storage

[Guspaz]

I'd rather a drive fail suddenly without warning if the alternative is "slowly and silently corrupting data until somebody notices it because it tripped a SMART threshold". Because I've had that happen, and all sorts of other nasty gradual failures.

Any enterprise setup is going to have redundancy built-in. If a drive up and fails suddenly and without warning, you swap it out, boom, done. If you've got hot or cold standby drives already in the array, icing on the cake. But if a drive starts a slow march towards failure, it can be a while before you or monitoring tools notice the issue, and even once you do, there's that temptation to say "Oh, it's still working well enough, we don't need to replace it quite yet."

Re:depends if you are IO bound or need storage

[hawkinspeter]

Generally, RAID1 needs the disk sizes to be the same and usually the writes have to be committed to both drives, so you'd most likely end up with a RAID that is limited to the size of the SSD and limited to the speed of the HDD.

Rather than using RAID, you could use the SSD as a single drive and have differentials/backups copied across to the HDD every couple of minutes or so - that'd provide some protection against disk failure without overly penalising the speed.

Re:depends if you are IO bound or need storage

[hawkinspeter]

I'd agree with that. I prefer systems that fail fast rather than fail slow - they don't leave you wondering if things are working ok.

Re:depends if you are IO bound or need storage

[WhiteDragon]

I'd rather a drive fail suddenly without warning if the alternative is "slowly and silently corrupting data until somebody notices it....

What about SSD "Time Warp" type errors?

Re:depends if you are IO bound or need storage

[Anonymous Coward]

This may be true for retail or manufacturing, but for any tech/science/engineering business that relies on data, 1 TB is a trivially small amount of information.

Re:depends if you are IO bound or need storage

[TheLandyman]

weak sauce.. 500 tps avg, 100 million inserts/day (between 00:00 and 07:00). Oracle 10g RAC on 4 AIX lpars with a DS8300 san.

Re:depends if you are IO bound or need storage

[Doctor Memory]

HairyFeet's customer however, lost a load of work when their drive broke without warning.

HairyFeet stated the customer was a gamer. I doubt they had a lot of "work" on their machine.

Re:depends if you are IO bound or need storage

[The Archon V2.0]

And a lot of empty space.

Aww, so many things you could do with the empty space! Like fill it with a bunch of pornographic photos, except with Hitler's face clumsily photoshopped over the women's. Make 'em pay for their curiosity with short-term impotence.

Re:depends if you are IO bound or need storage

[bobaferret]

:) I don't even begin to think we are a really high traffic site. But we do fall into the category of I/O bound vs. space. SSDs have enabled us to keep up with our demand while keeping our infrastructure costs down. As well as the amount of infrastructure we have is able to be smaller as well, which means the number of staff required is smaller.

Why wouldn't you run with DB2 on all of that IBM equipment?

Re:depends if you are IO bound or need storage

[swb]

I wonder why there isn't either a single unit disk drive that does this automatically -- acts as an SSD and mirrors all writes to the HDD.

Also nice would be a SSD aware RAID controller that would allow an intelligent RAID-1 mirroring between an SSD and a mechanical disk or three-way mirroring (SSD mirrored to HDD mirror), with the intelligence and cache to make all the HDD writing transparent to the system. At a larger level would be a smart storage system that kept frequently accessed blocks on SSD for reads, some capacity for writes but kept it all mirrored to HDD for reliability.

Since HDD capacities are so much greater, an even better concept would be a hybrid disk whose capacity was its mechanical size, but with some kind of intelligent caching system that used, say, 75% of an SSD for the most commonly read disk blocks and 25% for writes but mirrored the writes to the mechanical disk.

This way you get all the "reliability" of a mechanical disk with the speed of a SSD.

Re:depends if you are IO bound or need storage

[hairyfeet]

When I say "don't have time for backups" I mean they don't have time for DAILY backups which I wouldn't settle for less if they were running SSDs. There have been a few who haven't listened to me (which I tell them 'Don't come crying if you lose all your stuff') but most have followed my advice and made backups of important data their mantra. Some of the poorer ones use a cheap flash stick to keep their pictures, most have let me pick them up an external drive, but nearly all have taken my advice.

Oh and laptops? I try to steer folks away from those POS piles of plastic. I tell them "buy the cheapest junker you can because it doesn't matter, it'll be dead in 3 years or less regardless" and tell them to take the money they save and let me build a good solid multicore desktop, which will easily last them for many years. Just the other day I had the checkout girl in the local grocery ask me if I'd come out to her car to look at a PC given to her by a friend to "see if you can upgrade it a little" cause the poor thing had two kids and a checkout girl salary and I nearly fell over laughing when I saw what she had. When it looked like she was gonna cry I said "Oh I'm not laughing at you honey, don't cry, I was laughing because I built that old tank nearly a decade ago out of spare parts at the shop. On the bottom I wrote "Frankenputer" in marker which I showed her. I sold her a late model P4 with board, RAM and threw in a CD burner for $60 with install included and old Frankie is still going after all these years.

As for HDDs failing? I don't know, maybe I'm lucky because I'm half a day from the Newegg warehouse, who knows, but I honestly have seen a drive "just fail" in years. It has been so long I can actually remember my last two failures here at the shop. The first was the infamous IBM Deathstars which a local insurance company was unfortunate enough to get some HP desktops with 'em, and the other was the circa 2004 Maxtor DiamondMax which it turned out they used an early "green" solder that would break after repeated heat cycles.

But other than that knock on plastic I haven't seen a single unexpected failure. Sure drives have died, but I've always gotten plenty of warning, be it SMART, or the drive getting noisy, or Windows delayed write failures, something. maybe because I always ensure the case has good airflow?

And sadly I agree with ya on the RAID 0, but what could I do? If you've ever dealt with a gamer customer they are all "must have teh benchmarkz LOL!" with more money than sense. I mean his grandmama is running a fricking Skulltrail box because that is the SLOWEST hand me down the guy had! He is now on the top Intel chip (i970? He is the ONLY customer I'll buy Intel for since I switched my shop to AMD, only because he and his cousin have been buying from me for over a decade) with THREE fricking Nvidia Fermi cards and over a thousand watts of PSU and 16Gb of RAM in that bitch. Like I said, more money than brains and if the man wants RAID 0 Raptors? That is what he is gonna get. I have it set up with 1 touch backup so when he crashes it does a backup so at least he's covered.

But after having the SSDs fail so quick (his Intel in 7 months, his cousin's OCD? OZD, OCZ? Hell I can't remember, it failed in 5 months) they didn't trust another SSD and frankly I didn't blame them. I still think the tech is too new, they have too many bugs that still need working out. It is like the early HDDs which were flaky and easily died. Give them another 5 years and hopefully the tech will be mature. But until then nuh uh, not gonna do it. I pride myself on building machines that last (well except for the gamers who won't listen to anything but the benches, but they rarely keep a machine longer than 16 months before moving on) and SSDs are just too flaky ATM.

Re:depends if you are IO bound or need storage

[hairyfeet]

Well let your old neighborhood PC repairman help ya out with that bud. Here ya go, a little gem called HD Tune which you can install (or put on a flash as a portable if you like) which will not only give you an error scanner but with also give you a full SMART report including ANY changes that have occurred. Quite useful for spotting those problems loooong before they become a problem.

if you'd like it along with many other useful tools you might want to try Kel's CPL Bonus Pack which gives you not only HD Tune but over a dozen great tools right in the Windows CPL, like the HP USB bootable format tool, CPU/GPU-Z, one click driver backup, just a ton of nice features. It works on everything from Win2K-Win 7 X64 BTW, it'll just put them under 32 bit control panel on the X64 OSes but otherwise work perfectly.

And finally if you have a 4Gb or greater flash lying around (and what geek doesn't?) then I'd suggest computer repair utility toolkit v2. Now it is about 2 years old since the FOSS guys had a shitfit and made them take it down, but since this is the net and nothing ever disappears here you go. Simply update the apps and voila! A full toolkit that does just about everything, hardware, software, malware, tweaks, you name it, all with a nice launcher (which is easy to add your own apps to) on a stick.

Anyway i've found with the right tools NO HDD failure should frankly be unexpected...well unless you drop it of course. Otherwise with the above you'll get a heads up weeks in advance which gives you plenty of time to hit Newegg and get a replacement. Enjoy!

Re:depends if you are IO bound or need storage

[hairyfeet]

Hell I've been thinking this for years and it wouldn't even have to be that "smart" of a storage system, just have the SSD as the first partition with the drive making a differential in a hidden partition on the HDD with failover so if the SSD goes tits up it would just function as a HDD.

To me the perfect drive would be thus...128Mb ultra fast RAM (probably DDR 3) as a cache followed by 120Gb-240Gb SSD followed by 2Tb onboard HDD. The fat cache would allow for intelligent disk queuing while helping to keep the HDD writes from bottlenecking the SSD, as I said a hidden partition with failover would keep you from worrying about SSD failure, and with 1.5Tb (after removing the hidden partition space) you'd have plenty of room for your non OS files, all in a convenient 3.5 or 2.5 form factor.

Re:depends if you are IO bound or need storage

[Anonymous Coward]

I can confirm this. I've had many drives just suffer electronic failure and not show up in the BIOS any more (this was especially prevalent on WD drives). Each of these cases had no prior warnings at all.

Re:depends if you are IO bound or need storage

[swb]

Why stop at 128MB RAM? With RAM densities as they are, why not 4 or 8GB (or more) of RAM?

With RAM->SSD->HDD, for most operations in many systems you'd quite often be running as a RAM disk. The RAM->SSD times should be fast enough on writes to alleviate a lot of panic about cache loss in a power outage.

Re:depends if you are IO bound or need storage

[Guspaz]

That's a specific bug with certain consumer-grade OCZ drives. I have had the exact same problem happen to me with regular HDDs due to a bug in the filesystem driver. Admittedly, it was a different cause, but the same effect. As far as I know, Intel has never suffered from that issue with their drives, nor have any other enterprise-grade drives.

Intel has had a few issues with their drives, but they've thankfully been all of the immediate-failure type that works best with RAID.

Re:depends if you are IO bound or need storage

[hairyfeet]

Because if you make the cache TOO big you're gonna have to provide a battery backup in case of power loss? In AR we reached 119f with the heat index (113f actual) and the handyman was up until 4:30AM replacing blown fuses and breakers (building est 1923, actually used to be a hotel on the Chitlin circuit) So I know about power losses.

So when you figure in the cost of a battery powerful enough to feed 4-8Gb of RAM as well has the HDD motor long enough to flush the cache in case of power loss you're talking a drive that quickly becomes crazy money With 128Mb of fast cache and an ARM CPU onboard a couple of watch batteries would be enough to feed it long enough to do a shutdown on loss and of course the SSD wouldn't have to worry about power failure, so you'll only have to dump 128Mb to the HDD before shutdown. With 128Mb you could dump the whole thing in a single rotation in a "scratch" file that is loaded upon restart.

Is this a Slashvertisement?

[MikeTheGreat]

I'll be honest - I didn't really RTFA that closely, in part because it just fawns over the SSDs.

Can someone tell me why this is significant? (Because it's EBay, because it's the first large-scale deployment of SSDs like this, etc, etc)?

Thanks in advance (and sorry about the clueless SSD noob posting :) )

Re:Is this a Slashvertisement?

[smash]

essentially because SSD has far better IOPs, you need less units to get the same speed. So you can cut the size of the storage array in half. Ebay are clearly io bound rather than needing huge storage space. So for them, its a win.

For others, maybe not so much.

Re:Is this a Slashvertisement?

[Anonymous Coward]

Seems like it's no different than the story about AOL from last year, just twice the storage. But I'm sure AOL has upgraded some.

Re:Is this a Slashvertisement?

[erroneus]

I don't know, but I would be checking ebay for a butt-load of cheap 15k fiber channel drives for sale there.

Re:Is this a Slashvertisement?

[Anonymous Coward]

I don't know, but I would be checking ebay for a butt-load of cheap 15k fiber channel drives for sale there.

Awesome. I'll have to contact the vendor to see if they'll agree to accept payment on the side via google checkout.

Re:Is this a Slashvertisement?

[tbannist]

It's not significant at all, they're moving from FibreChannel drives which are notoriously small and expensive to SSDs. My last employer had a FibreChannel disk array. If I remember the prices, we were faced with a choice like 240 GB FibreChannel for $1000 each or 2 TB IDE drives for $100 each. It was obvious that moving to anything that wasn't FibreChannel was a good idea, because for the same price we could get either 480 GB of FibreChannel or 40 TB of IDE.

Re:Is this a Slashvertisement?

[afidel]

The pricing from most vendors is more like $1,000 for a 450GB 15k FC drive or $500-600 for a 2TB NL SAS drive, the FC drive will provide ~7x more worst case IOPS and have 1/3rd to 1/2 the annual failure rate. It really depends on if you need more IOPS or more storage as to which you use. We've found that by using a large striping array our workload tends to meet a balance between IOPS required and storage space about where the 450GB FC drives are. Our next array will probably contain a mix of SSD's, 15k FC drives, and NL SAS drives with intelligent array software to move things around between the tiers and using the SSD's for cache.

Re:Is this a Slashvertisement?

[jedidiah]

This only works out if you have lots of wasted space in your array.

This still comes down to why you would have used SSD 10 years ago before most slashbots ever even heard of the tech: You simply need the IOPS regardless of cost.

I could see eBay having this requirement. The rest of us not so much.

Re:Is this a Slashvertisement?

[jedidiah]

Yeah. Had forgotten about that part.

"Enterprise" drives tend to be small and expensive to begin with. Moving from one variant of this to another is probably not such a big change after all.

Re:Is this a Slashvertisement?

[Anarke_Incarnate]

There's a huge difference between the Nimbus and Violin systems. Violin has a method by which to keep the I/O flat and not have to suffer a write cliff. Nimbus doesn't appear to have the same features, though if you care only about density, it seems good.

Eve Online

[Anonymous Coward]

Eve Online went this route too for their blades.

Wonder why not 2.5" SAS drives..

[RightSaidFred99]

They're expensive, but nowhere near as expensive as SSD. I guess if performance is that important, it makes sense but how many Ebay/Google/Amazon situations are there out there?

Re:Wonder why not 2.5" SAS drives..

[fuzzytv]

Because you won't get the IOPs or speeds you get with SSDs? SAS driver are still the traditional drives, so the random access is a pain.

Re:Wonder why not 2.5" SAS drives..

[Anonymous Coward]

SAS and SSD are not mutually exclusive.

Re:Wonder why not 2.5" SAS drives..

[Anonymous Coward]

Don't know for sure, but enough to keep enterprise SSD providers like Nimbus and Violin and TMS in business. And still, you don't need to be big to need speed. We're very small potatoes in the scheme of things, but we have enough ram on our servers to entirely cache our important indexes. The speed is worth the money.

Re:Wonder why not 2.5" SAS drives..

[beelsebob]

Because you're talking about random access rates of a few hundred mess per second vs 10 mess per second if you're lucky seeking all over on a platter. The number of I/O ops an SSD can get through compared to even a fast HDD is insane.

Re:Wonder why not 2.5" SAS drives..

[bertok]

Because for small random I/O operations, SSDs are about 100x as fast as mechanical drives, but nowhere near 100x as expensive. This is not too dissimilar to moving from spools of tape to hard drives. Tape still has better capacity per dollar than disk, but you don't see anybody booting their PC off tape, do you?

You don't have to be a mega-corp for the speed advantage to make sense, which is why usage is increasing across the board.

For example, I recently installed a relatively small terminal server farm, and one of the performance tests I did revealed that for typical business application usage it's just not possible to utilise much more than 30% of the CPU and memory capacity of a server blade using mechanical drives!

Server OEMs like HP are already shipping blades with SSDs as a standard option, and I've seen SSD storage arrays in production usage. The time is coming very soon now when "storage" in the enterprise will mean solid state only, and spinning media will be used for the same role as tape is now: archival and backup.

Re:Wonder why not 2.5" SAS drives..

[TheRaven64]

Small SAS drives have really small platters, which means that their seek time is somewhere around quarter to half the seek time of a cheap drive. This means they get 2-4 times the number of seeks per second, which means 2-4 times the number of I/O operations per second (IOOPS). For a site like eBay, where people browse fairly randomly over their entire site, the IOOPS number is the most important. A cheap SSD gets 100 times as many as a cheap hard disk. More expensive ones get 1000 times as many. Unless the cheap SSD is 25 times as expensive as the SAS drive, then it's a better choice. Even then, one SSD requires a lot less power and cooling than 25 SAS drives.

I'm slightly surprised that they didn't go with something like ZFS's L2ARC, with maybe 20TB of flash and 100TB of slower disks. I'd have thought that the majority of eBay's disk accesses would be contained within a subset of their data (most people look at new listings and ones that will expire soon), so a big flash cache would give them almost the same performance at a lower price.

Re:Wonder why not 2.5" SAS drives..

[FreakyGreenLeaky]

Exactly. We use SSDs wherever performance is needed and once you (and your customers) have experienced the gains, there is simply NO going back. Even for a small accounting database server, using SSDs provides compelling gains - eg, for batch runs, instead of waiting half an hour, the wait is now something like a minute or two. The traditional HD will end up as an archive medium eventually... Seagate better get their shit together or risk becoming another SGI, SUN, whatever.

The gains far outweigh the costs.

Re:Wonder why not 2.5" SAS drives..

[Charcharodon]

Because for small random I/O operations, SSDs are about 100x as fast as mechanical drives, but nowhere near 100x as expensive. This is not too dissimilar to moving from spools of tape to hard drives. Tape still has better capacity per dollar than disk, but you don't see anybody booting their PC off tape, do you?

Just had a flashback of my TRS-80 that I had as a kid. Pressing play on the tape drive to listen to the data so I could line up on a program to load. Cassett tapes were expensive so you put lots of programs on them and had to write down the counter number in order to fast forward to them.

Seek time could be measured in minutes if my little sister or parenst were bugging me.

Re:Wonder why not 2.5" SAS drives..

[EraserMouseMan]

Seagate, Maxtor, Western Digital, etc will fade slowly into extinction much like the 3.5" floppy disk. Heavy competition from SSDs will force these companies to consolidate together just to stay alive. Eventually nobody will want to buy a mechanical drive anymore. Mechanical HD companies will not begin manufacturing SSDs because they are not set up at all to fab silicon. Only the RAM & CPU shops are. And, surprise, look at all the SSD manufacturers they are yesterday's RAM manufacturers.

Re:Wonder why not 2.5" SAS drives..

[drinkypoo]

I'm slightly surprised that they didn't go with something like ZFS's L2ARC, with maybe 20TB of flash and 100TB of slower disks.

So do you suggest they go with Oracle, which is fucking evil, or with FreeBSD, which implies a whole FreeBSD support infrastructure inside your organization?

a big flash cache would give them almost the same performance at a lower price.

ARRRRRRGH I WANT DM-CACHE NOW ARRRRRGH. Urgh. Sorry, involuntary reaction.

Re:Wonder why not 2.5" SAS drives..

[jedidiah]

No, not really.

Cost will always be an issue. There has also been a movement in the industry to support "larger and cheaper" storage. The idea is to have a "tiered" system where you can put your stuff on storage arrays with variable costs depending on your requirements.

The "archive" use case is actually a pretty big one.

Re:Wonder why not 2.5" SAS drives..

[afidel]

$/GB is still about 1/10th for HDD's vs SSD so there will still be a market for spinning rust. We only have a few process nodes left before we start to hit some fundamental limits with etching silicon so simply shrinking features isn't going to get them to parity (and it's not like HDD manufacturers are sitting on their laurels, they have a few more generations of their own already proven out in the lab). Also any of the techniques that significantly improve the storage per chip area in SSD's also significantly decrease the average cell life. Because of these factors most enterprises will have a variety of solutions in their datacenter, each trying to optimize the IOPS/$ and GB/$ for the data being stored.

Re:Wonder why not 2.5" SAS drives..

[afidel]

Nah, they could have gone with EMC which has FAST VP for moving data between storage tiers and FAST cache which makes makes writes hit SSD first.

Re:Wonder why not 2.5" SAS drives..

[operagost]

Those guys all sell SSDs, you know.

Re:Wonder why not 2.5" SAS drives..

[Guspaz]

Even for home use, I figure nearly everybody scoffing at SSDs, thinking they're not needed, has never *USED* an SSD for their day-to-day computing. It's the kind of thing where you think "Nobody needs that", and then you try it, and it turns out the be the single biggest performance upgrade you can make to your computer. And then you get used to it, and think "It's not that big a deal, HDDs weren't so bad", and then if somebody takes away your SSD, your computer feels like molasses.

A while ago, I built a new computer for a friend of mine. Got a great deal on a consumer SSD for him too. Later, he got a new laptop, and was complaining to me about how slow it was. After some prodding about what was slow about it, I realized that it wasn't the CPU, or the GPU, it was that he was now used to his desktop's SSD and going back to a notebook HDD was too big of a performance hit for him.

Re:Wonder why not 2.5" SAS drives..

[Guspaz]

The thing is, the cost-per-gig will eventually not matter, because the cost-for-enough-gigs will be sufficiently low. The point will come where people can get an SSD that's "big enough" at a price that's low enough. It won't matter that you'll be able to buy a 5TB drive for the price of your 1TB SSD, because the 1TB SSD will be big enough for the average person.

Re:Wonder why not 2.5" SAS drives..

[afidel]

There will still be a market, tape still exists even though there is essentially 0% consumer use of tape. Heck with tape there are actually more manufacturers than there are in the HDD business.

Uhh.. cost?

[Dahamma]

Of course everyone would love to replace all of their storage with SSD if price was no object.

The closest they come to mentioning cost is:

Though SSD is typically a magnitude of order more expensive than hard disk drive storage, Craft said the Nimbus arrays were "on par" with the cost of his previous storage, which a Nimbus spokesman said was from NetApp and HP 3PAR. (Craft declined to identify the vendors).

So, cost of new SSDs was similar to whatever HDDs they bought years ago? Yeah, that's kind of how it goes...

Nimbus prices its product on a per-terabyte basis - it charges $10,000 per usable terabyte

$10,000 per terabyte. Ok, then. Sure, it's faster, if you are willing and able to pay 10x the cost of *current* HDD-based systems...

Re:Uhh.. cost?

[peterbye]

What's 'a magnitude of order'?

Re:Uhh.. cost?

[Anonymous Coward]

Depends on your workload. (Disclosure: I work in storage for a living.)

Sometimes, what you need is raw, bulk storage. There are two serious contenders in this space: tape, and disk. You use tape if you have a lot of data you need to store, but not much that you need to access regularly: less power, and it scales to near infinite levels of storage (at the cost of very slow access for a given piece of data.) Or you use disk if you need to access most of it reasonably regularly. SSDs are not, and never will be, a contender in this space - you're paying through the nose on a per GB basis.

On the other hand, sometimes what you need is IOs per second. Database administrators are very familiar with this - you need a bit of data from over here, and a bit of data from over there, and maybe a little bit more from somewhere in the middle, and you need it five minutes ago. Traditionally, you got this performance by building a large array across many spindles, and giving up half, three quarters, or even more of your disk space, in return for that nice, fast section of disk on the outside of the platter. Lots of running hard drives, drawing lots of power, generating lots of heat, and costing a lot of money for storage space that isn't even half used - because if you throw something else on that empty space, you completely ruin the throughput of your major production database.

In that latter space, SSD is king. Sure, it's more expensive on a dollars per GB basis, but hey, guess what - GB isn't the important metric here. You figure out which bit of data is being hammered, and you move it across to the SSD. Rather like profiling an application: pick the function that takes 90% of the time in the software, optimise the wazoo out of it, and you get a significant improvement (rather than picking something at random and optimising it to billy-oh, and getting not much return for your investment.)

So yeah - SSDs aren't going to compete in raw capacity any time soon. But in random I/O performance, they make a hell of a lot of sense. In some respects, yes, they most definitely are cheaper than traditional platters of spinning rust - see the aforementioned massive RAID set across dozens of spindles.

Re:Uhh.. cost?

[SecurityTheatre]

Keep in mind, they're likely running parallel RAID configs to maximize throughput, meaning something like RAID 1 or RAID 5-1 or some other proprietary method that may provide only 1/3 or 1/5 the usable capacity in order to get the throughput required.

In fact, since these drives can theoretically be about 50x faster in random-seek reading of data, you could actually say that per the throughput AND size, they're actually cheaper than a comparably capable array of spinning discs.

Re:Uhh.. cost?

[smash]

Again, depends if you need bulk storage, or fast i/o. If, to get the IO throughput required, you need to purchase far more hard drive spindles than you otherwise would need for the capacity required, then the total amount of storage you get being less with SSD may not be an issue (so long as it is "enough"). e.g. (fake numbers), if you need say 100k iops and 1tb of space, this could perhaps be done with 10 magnetic disks, or 2 SSDs. the additional space provided by the magnetic disk is of no use if your only reason for purchasing multiple disks is to gain IOPs.

Many big corps like ebay are likely far more concerned with gaining higher throughput than more storage space.

Re:Uhh.. cost?

[TubeSteak]

$10,000 per terabyte. Ok, then. Sure, it's faster, if you are willing and able to pay 10x the cost of *current* HDD-based systems...

There will always be applications where the high dollar [latest and greatest] solution will provide *vastly* better performance than the current 'standard'.

If you're replacing 8 or 9 TB of 15K drives with 1 TB of SSDs, then the accounting becomes a bit more manageable.
And raw price:performance doesn't always tell the whole story, not when paying more can save in other ways.

Re:Uhh.. cost?

[sheddd]

10x

Re:Uhh.. cost?

[Anonymous Coward]

If it is 10x more expensive, but 100 times faster, as long as you are IO bound, not space bound; then it make sense.

Re:Uhh.. cost?

[Anonymous Coward]

The opposite of an order of magnitude?

Re:Uhh.. cost?

[Anonymous Coward]

$10,000 per terabyte. Ok, then. Sure, it's faster, if you are willing and able to pay 10x the cost of *current* HDD-based systems...

Are you smoking crack? You can get 120gb SSD drives for $160-$260 these days, that comes out to around $1333-$2300 per terabyte. You also have to take into account that one SSD is faster than most RAID setups. It is also much cheaper to run SSDs in database servers than it is to use regular hard drives in a RAID array. Not to mention the savings in power use. Tom's Hardware has a good writeup on this.

http://www.tomshardware.com/reviews/ssd-reliability-failure-rate,2923.html

Re:Uhh.. cost?

[qxcv]

Singular of "several magnitudes of orders".

Re:Uhh.. cost?

[Anonymous Coward]

SSDs are not, and never will be, a contender in this space

Bullpoop, of course they will be - eventually.

Re:Uhh.. cost?

[Charcharodon]

1 TB can be had for about $1,675 these days. Take a look at OCZ SSDs on new egg. The price is falling nicely, though I doubt it'll catch spinners in the next 10 years for cheapness.

Re:Uhh.. cost?

[bareman]

If I had a mod point, I'd mod this response upwards.

The parent to it was considering only capacity. And from a capacity standpoint, no SSD doesn't make any sense and is about 10x the price. But for performance SSD is king and yes, cheaper than the spinning rust.

Re:Uhh.. cost?

[Anonymous Coward]

no, 10x would be an order of magnitude, not a magnitude of order!

Re:Uhh.. cost?

[LordLimecat]

You figure out which bit of data is being hammered, and you move it across to the SSD. Rather like profiling an application: pick the function that takes 90% of the time in the software, optimise the wazoo out of it, and you get a significant improvement (rather than picking something at random and optimising it to billy-oh, and getting not much return for your investment.)

Or you do what eBay is apparently doing and say, screw it, we're doing 5 blades, and throw all of your storage on SSD,

Re:Uhh.. cost?

[LordLimecat]

What are the failure rates? That seems rather relevant when youre dealing with massive RAID arrays.

Re:Uhh.. cost?

[fwarren]

I have to agree. There is even the possibility that with only 1 hard drive manufacturer left in the game with a limited R & D budget that gains in storage capacity on hard drives will slow down. Not to mention limited production runs will increase the cost of the media.

Whereas SSD's will become more commonplace, the cost of production will fall and capacities will rise. As long as some form of optical media does not come along and make SSD's obsolete. SSD's have a bright future in front of them.

Re:Uhh.. cost?

[neokushan]

x10?

Re:Uhh.. cost?

[drinkypoo]

SSDs are not, and never will be, a contender in this space - you're paying through the nose on a per GB basis.

This is one of those predictions guaranteed to fail. Eventually it will be cheaper to produce SSDs than disks with a bunch of moving parts. How eventually? Too eventually for my tastes... but someday.

Re:Uhh.. cost?

[neokushan]

I'm not so sure, it all depends on moore's law and how well we can keep up with it. Essentially, SSDs should go the way processors have - doubling in capacity every 18months or so.
For the sake of argument, let's say that a 120Gb SSD is $200, the same $200 in 18months time should, theoretically, get you 240Gb. In 3 years time, it should be 480Gb, 4.5 years would be nearly 1TB and in 6 years, 2TB SSDs shouldn't be that unthinkable, for the same $200 you'd spend today. It was only couple of years ago that a 2TB Mechanical HDD cost about that and I believe it's a lot harder to increase the capacity of HDDs than it is to perform die shrinks. Then when you factor in that the value of Mechanical drives is going to plummet as the capacity difference between them and SSDs narrows, there'll probably be a lot less investment in them in future. Of course I could be just making this all up, it's just my own observation.

Re:Uhh.. cost?

[tompaulco]

Agree. So in other words, if we read between the lines, if they had replaced their older spinning disk with new spinning disk, they would have reduced their rack space by 20 times.

Re:Uhh.. cost?

[Guspaz]

$10,000 per terabyte. Ok, then. Sure, it's faster, if you are willing and able to pay 10x the cost of *current* HDD-based systems...

I guess you missed the part where they said that the Nimbus pricing of $10,000 was "on par" with the HDD-based storage arrays they had from NetApp and HP before? Fibre-channel HDD pricing is in the same ballparks as enterprise SSDs.

Re:Uhh.. cost?

[Anonymous Coward]

I think in this story's case, it's redundant across multiple U's.

Re:Uhh.. cost?

[Anonymous Coward]

and you move it across to the SSD

A lot of PHBs have learned that this sort of pick-and-place DBA activity carries risks and hidden costs. Adding additional tiers to your storage increases the complexity of your system which crops up in surprising ways. It is entirely plausible that one may conclude the higher cost equipment produces the lower overall cost and/or risk.

Re:Uhh.. cost?

[Dahamma]

That wasn't my statement, that was a quote from TFA. And I was pointing out that it doesn't sound "on par with HDD solutions" to me...

Re:Uhh.. cost?

[Dahamma]

Eh, no, I quoted that IN MY POST... :) And that was the point - their old HDD system cost as much (and took up more space) as the new SSDs. Heard of Moore's law?

And for large storage solutions SSDs are still significantly more expensive. The do have plenty of advantages, but price is still not one of them...

Re:Uhh.. cost?

[geminidomino]

Fucking popunder scumbags...

What can I say? Old wounds can still fester...

Re:Uhh.. cost?

[cthulhu11]

RAID 1 isn't just about throughput - it's also about surviving drive / HBA failures. While I favor 3-way mirroring (eg. ZFS -- no HBA I've seen handles it), 5-way is serious diminishing returns, so I don't know where you get the 1/5 capacity figure. Note that it's not uncommon to mirror SSD's, and one can even get systems that mirror *RAM*. What you may be missing is short-stroking, where one deliberately only uses a fraction of each disk, limiting time lost to head seeking and optimizing on outer cylinders that have the most sectors/track.

Compact

[drmofe]

So the entire eBay VM operation could fit into 6 racks? 200 physical servers @ 1RU each = 5 racks 10x 10TB 2U SSDs = half a rack 5x 2U switches = quarter rack

Re:Compact

[bberens]

Ebay has 4 DNS entries visible from here, so I would safely multiply that number by at least 4.

Re:Compact

[Anonymous Coward]

So one night and a half-semi truck and all of e-bay can be stolen.

I'm in.

Zero details

[billcopc]

TFA reads like a thinly-veiled promo for Nimbus Data Systems, which I can only guess are pushing a Linux-based SAN appliance full of SSDs. Big whoop.

What I would love to know is: Why does eBay need 4000 VMs ?

Re:Zero details

[Konster]

They need 4000 VM's so they can buy 100TB of SSDs. :)

Re:Zero details

[Anonymous Coward]

they need 8000 VMs to make their overdynamic site fast enough. They have only 4000 because they are skimpy

Re:Zero details

[Monoman]

"We're not a storage team. We're Windows administrators who got into virtualization ... "

Re:Zero details

[Anonymous Coward]

Because it's the only way to handle that many simultaneous connections under windows, duh.

Re:Zero details

[Anonymous Coward]

What I would love to know is: Why does eBay need 4000 VMs ?

I used to work for a fairly small ISP. We had 400 VMs on VMWare, because we had to provide all the services an ISP provides: email in, email out, access to mailboxes, DNS caches, DNS resolvers, web servers, monitoring servers, RADIUS servers, LDAP servers, stuff for internal use, shell servers of various flavours, VoIP related servers, software load balancers...the list is a long one. Oh and double everything for redundancy and make some of clusters of up to 10 VMs for capacity.

When you're someone like eBay, you basically are your own ISP, and you still need to provide all those services to yourself. So I can easily imagine that when you're the size of eBay, having ten times that many seems fairly sensible.

Re:Zero details

[Threni]

> Why does eBay need 4000 VMs ?

They operate around the world, need fallover protection, protection against script kiddies, and presumably want to stress test performance improvement solutions. They probably also have development, test and preproduction versions of the about. 4000 VMs doesn't sound at all excessive to me.

Re:Zero details

[TheLink]

At my workplace we probably have at least 100 VMs already, if not more. We have more VMs than employees.

4000VMs actually sounds interestingly low if its a global total.

Re:Zero details

[TheViffer]

What I would love to know is: Why does eBay need 4000 VMs ?

Taking the scope of eBay, PayPal, or any of the other company under eBay and then adding up the total number of developers and quality assurance it should be very apparent why a system is required in which a VM can be created and deleted for testing and development purposes in a very short amount of time.

Time is money

Re:Zero details

[Doctor Memory]

Yep. We did some quick calculations a couple years ago when our service contracts came up for renewal and discovered that it would be cheaper to scrap a dozen of our oldest servers and just virtualize them on a new 16-core box with 64G of RAM. Since then, as machines have come off maintenance, we virt 'em up and host 'em on the megaserver. Worked especially well for several servers we kept around for support purposes -- they ran outdated software and were rarely needed. Now they're virtualized and we can spin 'em up, let the techs do their job for a couple of days, then spin them back down until next time. We just installed another megaserver, and I think we only have five physical "single-use" servers left. I think there are two that will always have dedicated hardware (licensing issues), but I imagine that within a year that's all that will be left. I already have dibs on one of the old racks...

Re:Zero details

[billcopc]

So are they testing 4000 versions of the code at any given time ?

I'm fine with a few hundred VMs, heck I have close to a hundred between my desktop and dev servers at home. Several thousand though, for a one-trick pony like eBay, seems gratuitous. Have they even changed much in the last ten years ? Growth, sure, but not many customer-facing changes.

Which kind of SSDs they've used?

[fuzzytv]

The article does not mention which kind of SSDs they've used, or have I missed something? That might be very interesting, especially when it comes to reliability. It's often claimed the SSDs are more reliable than traditional drives, but accrding to this http://www.tomshardware.com/reviews/ssd-reliability-failure-rate,2923.html that's not really true.

Re:Which kind of SSDs they've used?

[Anonymous Coward]

Reliable has several different meanings.
Reliable does need to mean that it doesn't fail. It could just mean that it fails in a more predictable manner.

Re:Which kind of SSDs they've used?

[justforgetme]

As far as I understand they used storage arrays from nimbus data

Re:Which kind of SSDs they've used?

[fuzzytv]

Yes, but Nimbus is not a SSD manufacturer. I'd expect them to buy a lot of SSDs and then build the storage arrays from them.

Re:Which kind of SSDs they've used?

[_Shad0w_]

I wouldn't be at all surprised if they just buy in bulk from a manufacturer and re-brand them, like Dell do with their drives. The last lot of Dell 2.5" SAS disks I looked at were just Seagate drives with a Dell sticker on them instead of the Seagate one (they didn't even change the model number).

Re:Which kind of SSDs they've used?

[Anonymous Coward]

Did you even read that article?

"Fewer devices installed means fewer devices to fail, too. Since you use one solid-state drive to replace multiple hard drives, consolidation ends up benefiting the business adopting flash-based storage. If the swap were a 1:1 ratio, that argument wouldn't work. But at 1:4 or more, you're really cutting into the number of disks that would eventually fail, and we can't let that point be under-emphasized."

Article Heading :-?

[sonamchauhan]

I got the impression ebay just terminated a hosting arrangement with Rackspace (the company) -- bringing it inhouse, and cutting Rackspace's revenues in half. :)

Re:Article Heading :-?

[Wizarth]

That is also what I read!

Re:Article Heading :-?

[suso]

<AOL>Me to</AOL>

Rackspace = company that sells VMs
rack space or maybe even rackspace = space you have available in your racks.

deployment time nine times better?

[rbrausse]

has improved the time it takes to deploy a VM from 45 minutes to 5 minutes

uh, any logical explanation for this? SSDs are snappier and the peak I/O can be faster compared to spindle drives - but not by factor 9, or?

Re:deployment time nine times better?

[rtb61]

At a guess comparing unformatted hard disk drives to SSD, sneaky.

Re:deployment time nine times better?

[vadim_t]

Yes, in the real world, data isn't laid out on the disk in the exact order it's going to be read. Especially when filesystem structures are involved.

Seeks are very expensive. I measured a desktop drive barely doing 300 KB/s when doing random reads. That's of course not very realistic either, but the real world performance is going to be somewhere in between that and the ideal contiguous read speed. Having a disk capable of 100MB/s managing only 5MB/s is quite possible.

Re:deployment time nine times better?

[Man Eating Duck]

uh, any logical explanation for this?

I wondered about the long deploy time on spinning drives as well. From the article:

The online retailer clones from templates on its storage to the VMs.

Even if you assume a very conservative 15MB/Sec write speed that indicates an image size in excess of 40 GB, which is enormous for a fresh OS instance. On would assume that it's stripped down to the bare essentials necessary for the task at hand?

Re:deployment time nine times better?

[owlstead]

Quite possible? That's not even hard to achieve, just unzip a few large files at the same time. I had huge problems with newsservers since PAR2, unrar and my own file operations (copying to the correct folder) were taking ages upon ages to complete. And that was *without* virusscanner active. The first time I unzipped a 180 MB Eclipse install on my SSD, I kept waiting for the prompt to come back to me - only to find out it was there all the time.

Re:deployment time nine times better?

[PipsqueakOnAP133]

Let's get some quick info about a WD Velociraptor. (http://www.anandtech.com/show/2508)
98.4MB/sec avg sequential.
4.6ms seek time.

During 4.6ms, you can do 463.5KB of transfer.
To do a 4K transfer, it's 0.04ms.

Now consider some sort of file lookup. What do you need to do?
You'll need to seek to the part of the disk which has your file system's tables, which probably points to a few tree nodes. This is equivalent to walking a 4 node linked list, I guess. Each of these would be a seek, and a 4KB read.
At this rate, you're spending 18.4ms moving the heads, and 0.16ms actually reading.
Afterwards, you can then chug along at your blazing 98.4MB/sec reading the file.... which might be just 128KB. (1.27ms)

Out of a total of 19.83ms, 1.27ms is actually spent doing your file transfer.

Going with a crappy SSD. Say 0.03ms seek time, and 98.4MS/sec sequential (so I don't have to keep typing in numbers into a calculator).
This means we're doing a 0.12ms for a seek, 0.16ms traversing the filesystem, and then 1.27ms reading the file.

Out of a total of 1.55ms... well, we're already 12.7 times faster than the hard drive, so uh, yeah.

So you're probably thinking, okay, but hey, this is for a random 128KB read. And we're trying to copy sequentially to another image, right?

Well, if you're doing multiple reads of the same template to multiple drives, and you just so happen to be doing these out of sync, then copying 128KB to each target drive will require a read from different locations, yeah? Which means every chunk (depending on how you schedule them) will incur this 12x penalty.

You could take advantage of caching. But caching won't work if your cache overflows before the next drive can use the already read data. So you'll either see caching as a win, or no help.

Of course, you can make bigger copy block sizes. This cuts the seek penalty down quite a bit. But then again, this 12x improvement in the hypothetical example is a hypothetically truly crappy SSD. Getting a Sandforce or Marvell controller gets you read speeds up 4x better than what I used to calculate this example.

A 9x improvement doesn't sound so unbelievable any more.

I bet Rackspace did not like it..

[Anonymous Coward]

The cut Rackspace by half - Did they like... auction off half of that company?

Reliability?

[Vectormatic]

I read a blog-post a while back stating that SSDs fail a lot more then you would expect. Somewhere around a year of heavy use seems to take most of the life out of a consumer grade ssd. Now i wonder how putting SSDs into Raid 5 (or 6, or whatever) will behave. If a certain model of SSD croaks around X write ops, then i think the nature of Raid will mean that your entire array of SSDs will go bad pretty closely together. It must suck to have two more drives go belly up while rebuilding your array after the first drive failure.

Perhaps it would make sense to stagger SSDs in different phases of their lifetime to keep simultanious failures at bay, use some burned in drives and some fresh ones.

Re:Reliability?

[Terrasque]

That's okay. I found a blog written by someone from the future, so I sent him a request to fetch some SSD durability data from 15 years into the future.

I expect a reply any minute now.

Re:Reliability?

[angryphase]

I read on the The Register that in fact these are far from consumer grade SSD implementations. Instead they are using specific set of flash module configurations:

The system uses 24 flash modules, not standard off-the-shelf solid state drives (SSDs), with different flash densities per module to provide the 2.5, 5 and 10TB capacity points.

Re:Reliability?

[MemoryDragon]

It really depends on the write cycles, the problem there is that you have about 3000 write cycles on a modern consumer grade ssd, combine that with the fact that most people buy really small ones to save some bucks and they start to use it heavily by swapping on them etc... and you have a situation where you very likely can reach those 3000s within a year or two.
If you use them in a sane manner and have a decent size so that wear levelling can do its magic you should not hit that limit within the lifespan of the computer itself.
But that is just consumer grade ssds, enterprise grade ssds are an entirely different game.

Re:Reliability?

[Vectormatic]

i fully realize that consumer grade SSD != enterprise SSD, but a tighter grouping of failures is still very likely due to the way SSDs work. I for one am curious to see what effect this will have on RAID strategies in the future. As for usage, people using their SSD for swap are either stupid, or like the speedup that much that they are willing to eat the cost of an SSD over a years time, but in enterprise land those drives can see very heavy use as well, especially when logging/databases are involved.

Also, shortly after posting the root post, i realized that theoretically rebuilding a raid array should only write heavily to the new disk and only read from the remaining parts of the old array, so triggering a second failure by rebuilding isnt that likely.

Re:Reliability?

[bug_hunter]

Well they are just replacing their VM servers, the databases are possibly elsewhere on the network so the writes to the SSD in that scenario should only occur when they update a VM. (Just guessing though).
Still, I take your point of a series of SSDs used for the same purpose are more likely to fail around the same time than ye olde HDDs.

Re:reliability?

[vlm]

i was under the (misguided?) impression that ssd's weren't, as yet, enterprise ready in terms of reliability?

People rationalize what they want. Can you believe there are people that claim Windows is enterprise ready, merely because they want to / have to use it?

Re:Reliability?

[vlm]

Perhaps it would make sense to stagger SSDs in different phases of their lifetime to keep simultanious failures at bay, use some burned in drives and some fresh ones.

Trust me, as I guy who's run raid arrays of spinning rust for well over a decade, you REALLY need to do that with old fashioned drives too.

Worst experience in the world is having a RAID-1 with two consecutive serial number drives and both bearings let go at the same time.

Re:Reliability?

[Threni]

Sorry, I lost you at `I read a blog-post`.

Give me five mins and I'll put up a blog-post somewhere stating that this is incorrect. I'll probably add some graphs too.

Re:Reliability?

[Vectormatic]

http://www.codinghorror.com/blog/2011/05/the-hot-crazy-solid-state-drive-scale.html

There, now read past the first line of my post and consider the point i was making about SSD failures spreadage Vs HDD failure spreadage and its effect on RAID strategies.

Re:reliability?

[bberens]

I was under the impression that the vendor practically gave away these racks at or below cost so they can run around saying eBay uses their storage racks.

Re:reliability?

[rubycodez]

The big iron vendors have been selling them to enterprises, failure just means rip and replace under warranty with no impact from users point of view, just occasionally replenishes the pool hot spares from the provided pile), data gets transparently restored from the redundant copies.

Performance or Price?

[mjwx]

'One rack [of SSD storage] is equal to eight or nine racks of something else,' said Michael Craft, eBay's manager of QA Systems Administration."

Is he talking about performance or price. I can imagine that a single rack of enterprise SSD's could easily cost the same as 9 racks of anything else.

Re:Performance or Price?

[FreakyGreenLeaky]

Performance.

Re:Performance or Price?

[bberens]

You're measuring storage capacity. Ebay is IO bound rather than storage bound. So they measure in reads/($*second) where $ is a function that incorporates comparative failure rates, electricity costs, and probably a few other bits that don't come to mind at the moment.

Re:Performance or Price?

[mjwx]

You're measuring storage capacity.

Actually I was being facetious.

Making allusions to the price of enterprise SSD's, 10 x more expensive then 15K RPM mechanical disks, at least in Oz.

Not going to argue about the IO benefits, because you're right.

huh

[Anonymous Coward]

so a drive with 1tb space is equal to 9 tb of anything else

BULLSH%T

Re:huh

[smash]

More like, the I/Os per second of 1 rack of SSD has 9x the THROUGHPUT of 1 rack of magnetic media. remember, we're talking about massive arrays here to get speed, not necessarily for disk capacity. If you only need the capacity provided by 1 rack of SSD, then being able to cut your rack space in half or less by getting the required IO in less drives can save heaps. Potentially, it can save you needing to build a new datacenter....which ain't cheap.

Re:huh

[Neil Boekend]

It's about speed, not capacity. They used to have 15K disks, which have a latency of around 2 ms. They switched to SSD's, which have a latency of around 0.1 ms.
Why does latency matter for them? They have large amounts of relatively small files (small images, item descriptions and so forth). These files are spread across the disks. Each file requested means the latency is counted. If a webpage needs 100 files the latency of the disk is multiplied by 100. Each user (and there are many at any given moment) needs 100 files. As soon as possible.
The used to solve this by having far to many disks. The amount of storage space may have been twice what they expected to need, because more disks means you can acces more files at once, effectively cutting your acces time. Now file 2 doesn't have to wait until file 1 is done, because it's on another disk (or actually RAID). This increases the speed, but at a cost.
Add to that the fact that SSD's have insane transfer speeds. An 15K rpm disk may transfer 300 MB/s, an simple SSD can transfer 550. Theoretically they can go much faster, but SATA can't keep up. These systems don't use sata, so their transfer speeds may be much higher.
IANAEOTS, so correct me if I am wrong.

Re:huh

[Anarke_Incarnate]

15K drives have a seek time of about 5ms. That is just to move the head before the data has started to flow. SSDs can get the data in microseconds.

Return On Investment

[ritcereal]

While most people instantly gravitate towards the upfront cost and performance of going solid state, I would make one important point. Reducing your data center space by 9 racks is significant in terms of power, cooling and that is all on top of the purchase price and support contracts. Regardless if ebay owns their own data center or colocates, the cost per square foot in a data center and the continued operation of such a large storage system is more then likely to provide a higher return on investment. eBay isn't in the business of looking cool and hip, they're in the business of selling stuff as cheaply as possible and I'm certain their CIO cares only about the bottom line.

Re:Return On Investment

[darkmeridian]

And don't forget that the SSD solution uses less electricity. That means you're not only saving on the electricity, but you're also spending less money removing the heat associated with using electricity. Being able to rebuild a server in five minutes as opposed to forty five minutes probably means they can keep more backup servers slack and turn them on only in case of emergency. Since SSD use less electricity on standby, they probably can keep the SSD idling. And if it's so much smaller, you can pack more of them in the same space, which makes your facilities more future-compatible; you don't have to get new space to expand.

Re:Return On Investment

[greed]

Less heat means you can also pack 'em in closer together: you don't need as much air circulating to keep things cool.

OTOH, I've used SSDs which ran pretty hot; they used more power than a 7200 RPM 3.5" disk, let alone a 7200 RPM 2.5" disk. If you're replacing 15k RPM drives, sure, that's a big power savings. But I remain unconvinced that 15k RPM is the right tool for many jobs--7.2k and striping works really well for me. (My group has no budget, so I have to do the best I can. The group with a budget--and the 15k RPM SAN--has I/O problems all the time.)

Re:Return On Investment

[Anonymous Coward]

Hard drives are already solid state. There is no vacuum, gas or liquid involved in a hard disk. It really annoys me to no end how phrases with precise technical meanings become buzzwords. Was it so hard to call it semiconductor drive? Awww, too many syllables? How about silicon drive? Better?

But but but it means "no moving parts"! Well then I guess my CRT monitor is solid state then!

78% power savings - that's pretty awesome too

[spineboy]

Might even pay for itself by the years end
Barring any major catastrophes - expect to see may companies with server farms to go this route soon

Re:78% power savings - that's pretty awesome too

[arglebargle_xiv]

Might even pay for itself by the years end

Given that an OCZ Colossus 2 (just under 1TB) retails for $2,500 (and even when converted to Canadian dollars that's still a lot), you'd have to be using some awfully expensive power generation for it to pay for itself by years end. Or were you referring to the SATA cables that you needed in order to plug them in? You could probably pay those off in power savings.

Re:78% power savings - that's pretty awesome too

[EraserMouseMan]

You guys are all missing the point. eBay missed the boat. They blew their whole wad on SSD failing to realize that GE just announced a new age of optical Nirvana.

Re:78% power savings - that's pretty awesome too

[Anonymous Coward]

No, it doesn't pay for much. An SSD consumes 1 watt or less. For simplicity, lets just say....zero watts. My older had drives consumed about 10 watts, but more recent ones (last 3-4 years) seem to have been 6-8 watts. For simplicity, lets say 10 watts. So a generous estimate is that an SSD will save you 10 watts per drive.

For a device in use 24/7, a good rule of thumb is that 1 watt costs $1 per year (based on a cost of 11.5 cents per kwh...your cost may be more or less). Right now the cheapest SSDs I've seen is $65 for a 64GB Kingston at costco. So even with our generous 10 watts saved, and assuming you can get away with this budget drives, You are still looking at over 6 years to break even.

Re:78% power savings - that's pretty awesome too

[bberens]

It's important to note that spinning disks have been around for a long time and the margin on them is small. SSDs are new tech so the margins are quite large (to pay for the research). It's highly unlikely that ebay paid the research premium and that the vendor for those SSDs cut them a huge deal for advertising purposes. It also doesn't take into account other costs such as life expectancy of the SSDs vs. spinning disks, whether the ebay CEO is on the board of the SSD manufacturer's company, etc.

Re:78% power savings - that's pretty awesome too

[nabsltd]

So even with our generous 10 watts saved, and assuming you can get away with this budget drives, You are still looking at over 6 years to break even.

Three to four years should break you even, since pretty much 100% of the power input for a spinning disk drive is turned into heat which must be removed by costly air conditioning.

Re:78% power savings - that's pretty awesome too

[GooberToo]

Not to mention the massive savings on rack space, which can get very expensive per square foot.

Chances are, their return is a year or two once you factor everything in.

Re:78% power savings - that's pretty awesome too

[jbengt]

Three to four years should break you even, since pretty much 100% of the power input for a spinning disk drive is turned into heat which must be removed by costly air conditioning.

Even fairly inefficient air conditioning removes 2 to 3 watts of heat for every watt input to the A/C. And decent air conditioining can be much more efficient in cooler weather.
So, if you're not talking about the capital costs of installing more A/C, it won't add that much to the operating costs.

Re:78% power savings - that's pretty awesome too

[Anonymous Coward]

Not to mention the massive savings on rack space, which can get very expensive per square foot.

Chances are, their return is a year or two once you factor everything in.

That's not relevant. The topic of this particular subthread is whether or not the electricity savings will pay for the drives.

Re:78% power savings - that's pretty awesome too

[GooberToo]

Nice troll AC. So you can start a subthread but you can't start a subsubthread which makes the original subthread moot?

Again....nice troll.

Yes!

[ThePhilips]

Finally we are getting a chance of seeing real reliability stats of the SSD!

That if eBay would be kind enough to publish the data couple of years later.

Re:Yes!

[crashumbc]

This is first thing that came to my mind...

If the SSD's only last half as long you may not be getting the ROI your looking for.

Re:Yes!

[ThePhilips]

I do not care about ROI of eBay - I want to see how SSD work in real-life compared to HDD.

As one never gets any meaningful information from vendors, the only hope is the feedback of the users. The user in the case is the eBay.

Re:Yes!

[Anonymous Coward]

Not likely. eBay, like where I work, doesn't really care. SSD's are expensive from a consumer perspective. To a big business, the cost is insignificant, even in light of how much eBay is saving in this example. They could fail 3 per day and no one would bat an eye. If you were to get a very cost conscious company that does nothing but storage (Backblaze as an example), they'd likely be very interested in those sorts of numbers and likely to publish their findings.

reliability?

[ushere]

i was under the (misguided?) impression that ssd's weren't, as yet, enterprise ready in terms of reliability?

VMs are in the IO category for sure

[Sycraft-fu]

You discover modern hardware does virtualization real well. You get a good host software, like vSphere or something on new hardware and you have extremely near native speeds. The CPUs handle almost everything just like it was running as the host OS, and sharing the CPU resources works great. Memory is likewise real good, in fact VMs only use what they need at the time so they can have a higher memory limit collectively than the system RAM and share, so long as they don't all try to use it all at once.

You really do have a situation where you can divide down a system pretty evenly and lose nothing. Let's say you had an app that used 2 cores to the max all the time and 3GB of RAM. You'd find that it would run more or less just as well on VM server with 4 cores and 8GB of RAM, half assigned to each of two VMs, as it would on two 2 core 4GB RAM boxes. ...Right up until you talk storage, then everything falls down. You have two VMs heavily access one regular magnetic drive at the same time and performance doesn't drop in half, it goes way below that. The drive is not good at random access and that is what it gets with two VMs hitting it at the same time, even if their individual operations are rather sequential.

It is a bottleneck that can really keep things from scaling like the other hardware can handle.

At work I use VMs to maintain images for instructional labs (since they all have different, custom requirements). When I'm doing install work on multiple labs, I always do it sequentially. I have plenty of CPU, a hyper-threaded 4 core i7, plenty of RAM, 8GB, there's no reason I can't load up multiples. However since they all live on the same magnetic disk, it is slower to do the installs in parallel than sequential.

If I had an SSD, I'd fire up probably 3-4 at once and have them all do installs at the same time, as it would be faster.

Re:VMs are in the IO category for sure

[complete loony]

For running on HD's I think there should be one OS with one picture of the filesystem. So that caching, prioritisation and request reordering can all occur in one place, with as much information about the immediate future as possible. Each VM should then import a section of that filesystem. But if you're talking about databases, they tend to replace most of the filesystem level caching anyway, so you'd still be better off with SSD's and / or dedicated storage.

Re:VMs are in the IO category for sure

[Anonymous Coward]

You make an excellent point on the scalability of disk vs. CPU/RAM. Latency is the real killer.

Regarding your parallel vs. sequential installs, wouldn't wouldn't subsequent installs catch up to the first one due to caching on the storage system? Or does it not have effective caching?

Once, I attended a class where the instructor had us fire off a disk-imaging program from our class PC's that all hit the same CD-ROM drive on a file server (the instructor's machine). After some initial thrash-seeking of the CD-ROM drive, the laggard machines caught up to the one in the "lead" due to already-cached data being available instantly and they all completed at the same time.

Re:VMs are in the IO category for sure

[badran]

This only works if your RAM/cach is greater than the data accessed. Which works great for CD, but not so great for multi-GB VM images.

Re:VMs are in the IO category for sure

[Anonymous Coward]

Yep, VMs are great, we use a couple hundred of them, but they're not for everything.

We moved our Splunk indexers back to native hardware because the VMs couldn't handle the I/O, it was killing them (this with Splunk ESS. they were holding their own without that, but couldn't take the extra workload.)

Re:VMs are in the IO category for sure

[Amouth]

so you went from multiple physical boxes with disks to a single VM host and single disk? that doesn't make any sense to me - except a failure in planning.

what we do for that is pass the disks from the san directly to VM - you get the benefit of VMs in reduced number of CPUs/RAM/Power and keep the same san requirements. then we combine VM's into groups to present on host disks on the san based on performance.

you can get the benefits of VM's without experiencing the performance problems of shared disks, you just have to plan for it.

An indication of mistuning?

[Anonymous Coward]

It might also might indicate that ebay has done mistuning of their environment.
Some idiots still think that for a web server you need to take the amount of total memory and divide it by the amount of ram one apache process takes. And then setting this value as the amount of apache tasks. Thus leaving too little 'empty' for filesystem caching. Sadly enough that mistuning advise was for years on the apache webserver faq.
Having enough memory available for filesystem caching is the number 1 priority to reduce IO operations. Only with large database servers which have completely random access, this might not help. And with web applications such as ebay it's rarely ever completely random.

But hey, it's ebay, and that they have a crappy infrastructure has been visible for years... running slowaris for a decade for SSL websites and such... absurd!
Any decent infrastructure should serve most of it's pages without any IO operation taking place at all. Just like Facebook and Google do. Disk is merely intended for persistant storage, but most gets served out of filesystem cache. But hey, the fact that they rely on 'virtual machines' already says enough about their level of intelligence...

SSD is however great for IOPs, as others indicate; especially oracle tends to do a lot of unpredicatble random IO which really gets a boots with SSDs... practical for small environments...
But I wonder what the ebay investment cost was, and what the performance increase would have been if they had invested that into RAM instead (without re-mistuning).

Greetings,
Jasper

They've got plenty of money, no wonder

[Kartu]

Took me a while to figure how much do they actually cut from the end price, it's not something they're pleased to tell you.
Especially with recent "adjustment of prices" at ebay, no wonder they have extra cache to waste on whatever idiotic idea comes to their IT management.
They get 8%+ of most below 500$/Euro items sold. Outrageous.

Re:They've got plenty of money, no wonder

[bberens]

Apple gets 30%. Really, 8% markup isn't bad to have your products on the shelf at a vendor like ebay is pretty good. Do you know the Amazon markup?

This is just great news!

[Max_W]

I hope my hosting company will change to such SSDs, and I will not have to wake up those rotating disks early in the morning.

They should clean up their HTML first....

[Anonymous Coward]

Have you seen the mess that is Ebay's HTML and CSS? They could reduce the size of the average page by 50% probably, if they hired somebody who actually knew how to code properly.

Forget it

[alphatel]

I call total BS on this post for a few reasons

1. 100 TB is about 250 SSD drives. Ebay runs a few hundred peta of storage over several datacenters throughout the US, not including all other countries.
2. We run several racks at Telx. Our cost per month in NYC including electric is only about 3k per cabinet. Does Ebay really care about 3 cabinets?
3. Over half the systems now access the shared storage which contains the drives? Yes, if I map a drive to a particular SAN I guess I am now accessing the data. Does that mean I am actually leveraging it?

All in all this is barely a dent in anything Ebay does. It sounds more like an experiment and hype of the drives they used.

Re:Forget it

[Sprinkels]

You are correct.

If you read the article closely, you will notice it is not about the productions systems, but the testing systems of the Quality Assurance Division.

It makes sense to use VM's for Q & A just for flexibility. If your production storage is about a few hunderd PB, 100 TB for testing seems obvious. When you run production on the scale of eBay, Doing Q & A work on just a server or 2 doesn't work out.

I suspect they need 4000 VM's for Q & A, for more than just simulating the production system. For instance they are likely to test on multiple EBay instances for multiple projects or versions of their own software. They might also need a lot of systems to test data replication. And of course they need to simulate a lot of simultaneous clients for performance and reliability test, you need systems for that too. When Q & A work is done on that scale you need to collect, store, and analyze huge amounts of testing data.

Of course when this works out fine, eventually the production storage will likely be equipped with SSD technology as well.

Re:Forget it

[Anonymous Coward]

I call BS on your self-elected "expert" claims on the internals of ebay's systems.

Re:Forget it

[Matheus]

RTFA: This is just the eBay QA division. (Ergo the need to rapidly deploy VMs all the time as they try different approaches and solutions). A point made in the FA is that due to the success they have seen in QA due to this switch to SSD eBay will most likely (positively) be pushing those changes out to larger parts of their business (such as the few hundred peta you refer to).

It is then that we will see how much eBay saves by changing over the big iron but one would think that the kind of successes they've seen in this relatively 'small' environment will scale quite well.

SSDs are still unreliable

[unity100]

even the most touted and expensive 'enterprise ssd' can die out on you unexpectedly.

Re:SSDs are still unreliable

[KillaGouge]

As can the most touted and expensive 'enterprise hdd'

Re:SSDs are still unreliable

[beanpoppa]

even the most touted and expensive 'enterprise ssd' can die out on you unexpectedly.

Yes. Unlike a magnetic hard drive. I often go into our data center, filled with 600 servers, and go listening for the hard drives that are noisier than normal so that I can predict which hard drives are going to fail next.

Re:SSDs are still unreliable

[slamb]

I often go into our data center, filled with 600 servers, and go listening for the hard drives that are noisier than normal so that I can predict which hard drives are going to fail next.

Dude. Work SMARTer, not harder.

only for reads.

[unity100]

SSD life is limited with number of write operations. you cant use them like normal disks in the business ebay is using.

but read operations are unlimited. so, if you are going to just read files from a hard disk, ssd makes the perfect candidate. in random reads, they are approx 40 times faster than best hdd at the minimum.

so, you just put 250 ssd disks, put your VM images on it, and, as the article says, it boosts your vm deployment time to other systems from 45 minutes to 5 minutes - there is nothing wrong with the article.

SSDs are faster in all respects.

[unity100]

not only random reads.

you have to use ssds for ALL reads, but, hdds for all writes. since ssd life is still limited with number of write operations conducted.

Re:SSDs are faster in all respects.

[Guspaz]

For SLC, which is what you'd find in enterprise, the number of erase operations (which is what has a lifespan, not writes) is a non-concern.

Recycle!

[trum4n]

I want the old hard drives! Please?

bought them on eBay??

[Anonymous Coward]

Hmmm did eBay buy their SSDs on eBay???

In Other News

[chrispatch]

Fibre Channel Storage equipment floods EBay!

LOAD "*", 8, 1

[Compaqt]

'nuff said.

Re:LOAD "*", 8, 1

[ajlitt]

e2... How quaint.

Deduplication + SSD

[snsh]

They sweet-spot in enterprise storage is doing deduplication with SSD with both direct attached and networked storage, plus 15k 2.5" and 7.2K 3.5" disk for the rest. Deduplication saves a lot space and with SSD it works like cache, especially in scaled out environments.

failure rates

[dna_(c)(tm)(r)]

From the paper "Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you?" by Bianca Schroeder Garth A. Gibson

In our data sets, the replacement rates of SATA disks are not worse than the replacement rates of SCSI or FC disks.

So the annual failure rates are apparently similar, regardless of vendor claims

Re:failure rates

[afidel]

Nope, MS, Google, and others have all release numbers which show SATA at an AFR of ~4.5% and FC/SAS at ~1.5%, NL SAS drive split the difference according to MS numbers. My own datacenter with about a thousand datapoints mirrors those numbers.

STDs

[Anonymous Coward]

Am I the only one who read the title as ebay deploys 100TB of STDs, cuts rackspace in half?

100TB SSDs?!!?

[sqldr]

Cuts Rackspace By Half

Yeah, and their bank balance. And I thought our £2 million SSD database cluster was expensive..

Re:Uhh.. cost? Answer: Not very much different.

[Anonymous Coward]

Yeh, if you compare enterprise MLC SSD and enterprise 15k rpm HDDs, the cost per gigabyte is pretty close.

480 GB enterprise SSD can be had for $1300
vs
146 GB enterprise SAS 15k rpm 2.5" at $400, 3 of these cost you $1200 to get you 438gb of storage

Of course you would probably get multiple SSDs and raid them for redundancy, but that is just an example.

Then if you factor in BOTH power and cooling savings you will see the SSD gets even closer or better for cost. There's a whole host of other savings involved in managing less disks, less space, less strain on AC, and AC failures are less of an emergency because you have so much less heat being generated.

1 SSD replaces multiple HDs?

[Firethorn]

One thing I haven't seen mentioned for the power savings is that in an extremely high random read environment where SSDs shine the most, you're generally looking at one SSD replacing multiple hard drives.

If you can get rid of 5 15k drives and replace it with 1 SSD, the capital cost difference is vastly reduced, if not eliminated, even before you look at power savings.