Slashdot Mirror
SSDs: The New King of the Data Center?
Nerval's Lobster writes "Flash storage is more common on mobile devices than data-center hardware, but that could soon change. The industry has seen increasing sales of solid-state drives (SSDs) as a replacement for traditional hard drives, according to IHS iSuppli Research. Nearly all of these have been sold for ultrabooks, laptops and other mobile devices that can benefit from a combination of low energy use and high-powered performance. Despite that, businesses have lagged the consumer market in adoption of SSDs, largely due to the format's comparatively small size, high cost and the concerns of datacenter managers about long-term stability and comparatively high failure rates. But that's changing quickly, according to market researchers IDC and Gartner: Datacenter- and enterprise-storage managers are buying SSDs in greater numbers for both server-attached storage and mainstream storage infrastructure, according to studies both research firms published in April. That doesn't mean SSDs will oust hard drives and replace them directly in existing systems, but it does raise a question: are SSDs mature enough (and cheap enough) to support business-sized workloads? Or are they still best suited for laptops and mobile devices?"
TL/DR: "The relative cost of the two configurations shows that over-all there are cost savings using the SSD instances"
at least for their use-case (Cassandra).
At work we also use SSDs for a couple terabyte Lucene index with great success (and far cheaper than getting a couple TB of DRAM spread across the servers instead)
By switching to SSD's on a data intensive web application, I got 20 times speed improvement - from 20 hits per second to 400. I trust SSDs more than physical spindles any day.
The question is really going to be what kind of shape the drives will be in a year or so from now after 12+ months of constant heavy usage. The usage profile in consumer computers is a lot different from that in a server, and the server workload's going to stress more of the weakest areas of SSDs. And when it comes to manufacturer or lab test results, simple rule: "The absolute worst-case conditions achievable in the lab won't begin to approximate normal operating conditions in the field.". So, while SSDs are definitely worth looking at, I'll let someone else to do the 24-36 month real-workload stress testing on them. There's a reason they call it the bleeding edge after all.
We have hundreds of SSDs in production servers. We couldn't survive without them. For heavy database workloads, they are the silver bullet to I/O problems, so much so that running a database on regular disk has become almost unimaginable. Why would you even try to do that?
You have to remember that enterprise level storage isn't a single set of drives holding the data, it's a hierarchy of different technologies depending upon the speed of data access required. Since SSDs arrived they've been used at the highest access rate end of the spectrum, essentially using their low latency for caching filesystem metadata. I can see that now they are starting to replace the small, high speed drives at the front end entirely. However, it's going to be some time before they can even begin to replace the storage in the second tier and certainly not in the third tier storage where access time isn't an issue but reliable, "cheap" and large drives are required. Of course, beyond this tier you generally get on to massive robotic tape libraries anyway, so SSDs will never in the foreseeable future trickle down to here.
Agrajag: "Oh no, not again!"
I paid extra to have two in each machine so I could RAID 1 them in case one died. Minecraft is write-intensive and we also had map generation, although the maps were written to magnetic disks because they are so huge. Before that year, we were running everything on magnetic disks on hardware RAID 1.
As soon as we switched, iowait went down to practically zero. System load followed. Map updates incurred high iowait on the magnetic disks, but had no impact on the SSDs or server performance, and they also finished a little faster.
I don't think we ever had an issue with an SSD going bad. We did lose a magnetic hard drive once. Because the Minecraft servers were on SSDs, we just reinstalled the OS on a new drive and mounted the existing SSD RAID. It went alright.
I would definitely use SSDs again if I was hosting something IO heavy, especially write heavy. They excel at it in a way that even a RAID10 of magnetic drives would be hard pressed to equal. You back up your data so if it fails, you're ready.
Perfect! Now we just to sit back and watch the price come down even further.
I am using Samsung (Pro) SSDs for all OS partitions and only use HDs for bulk storage nowadays.
Ok great but where can you find an affordable SAS SSD?
I really think that SSD have a great value but you must rethink you infra and application to work with them.
The enterprise class SSDs are not the same as the "consumer" ones: http://www.anandtech.com/print/6433/intel-ssd-dc-s3700-200gb-review
Don't be surprised if you stick a "consumer" grade one to a heavily loaded DB server and it dies a few months later.
Fine for random read-only loads.
And some consumer grade SSDs aren't even consumer grade (I'm looking at you OCZ: http://www.behardware.com/articles/881-7/components-returns-rates-7.html ).
Pricing really needs to come down on these things. A single drive can easily cost as much as a server, and when you're talking about RAID setups, forget it. It's still much more effective to use magnetic drives and use aggressive memory caching for performance, if you really need that.
Another 3 to 5 years this idea might have more traction for companies that aren't Facebook or Google, but right now, SSD costs too much.
I think that the wide range adoption of server SSDs also shows how far server installations have progressed toward eliminating all single points of failure.
In the passt HA and 'five nines' was something only done by a few niches, like telephony provider switches or banking big iron. Today it is common in many cloud installations and most sizeable server setups. A single component failing will not stop your service.
If your business can support the extra cost for the SSDs, a failing drive will not stop you and the performance of the service will see great improvements anyway. The power savings may even make the SSD not so costly after all.
SSDs are slow in that they rely on old school disk protocols like sata. Sure, you'll get better performance than spinning disk. But if you want screaming fast performance, you should look at flash devices connected through the PCIe bus.
Products from Fusion IO would be an example of this. Apple Mac Pro would be another: "Up to 2.5 times faster than the fastest SATA-based solid-state drive".
This is being driven primarily by increasing levels of virtualisation, which turns everything into a largely random-write disk load, pretty much the worst case scenario for regular old hard disks.
are SSDs mature enough (and cheap enough) to support business-sized workloads? Or are they still best suited for laptops and mobile devices?
I don't see maturity as a problem. If there is money to be made drive manufacturers will throw enough engineering and computer science talent at the task of solving the teething troubles. What interests me is that if SSDs mount a major invasion of server-rooms and data-centers worldwide it also means that we will now finally start to see SSD pricing drop like rock. Cheap high capacity external SSD drives, I can't wait. If we are lucky this will also popularize Thunderbolt with PC motherboard makers since that's where you start seeing some real performance advantages, i.e. when the time it takes to make a backup of your laptop/desktop system to an external drive drops by half or more compared to USB 3.0.
Only to idiots, are orders laws.
-- Henning von Tresckow
What a coincidence! I am getting ready to transition our main DB servers (couple of GB mysql data) to SSD, but I simply dont want to trust it that much yet. So my plan is to set up RAID-1, with an SSD and another conventional drive. There seems to be this "--write-mostly" option that tells linux to preferably read from the SSD. Anybody know if this is worth it? If it works? What kind of random access performance gains can i look forward to, running mysql on SSD? I found it surprisingly hard to find any good data on these subjects.
We've been using SSDs in our data centers for quite a while now, specifically to store our high volume i/o databases. Its pretty much an indispensable technology when your system depends on speed. Think communications & financial transactions. Delays can result in decreased user experience in the case of communications, or arbitrage opportunities for those who do it faster in the case of financial transactions.
I work for an Australian hosting company and we have deployed the SolidFire all-SSD SAN for our cloud-based hosting (shared, reseller, cloud/virtual server), the major benefits of an all-SSD storage solution speak for themselves: far lower I/O wait time, huge IOPS numbers - in SolidFire's case 250,000+ distributed IOPS in our current configuration. We've recently shifted from the HP SAS-based Lefthand SAN offering up to 15,000 IOPS to the new SolidFire all-SSD SAN and the team behind SolidFire are partly from the Lefthand operation from HP, so there's some good know-how there. The article is quite broad in its content, for big data applications SSD SAN storage is still incredibly more expensive ($/GB) than SATA or SAS based SANs - our SolidFire was a huge investment. Many hosting providers are now switching to all-SSD based servers for the performance benefits, however the drawback is primarily total storage capacity of course. For example a typical VPS node using local storage with 10 x SATA drives can get up to 4TB of usable RAID-protected storage. The numbers for an all-SSD node in RAID configuration would be much lower in capacity and suitable higher in cost. Its important to note that many people view SSDs as desktop only hardware, which is fundamentally incorrect, as there are many units out there that offer write longevity much longer than needed (5-10+ years). For many server based applications (not big-data purposes), SSDs are, and will become the predominant choice among many hosting companies. Not every provider can afford the investment of an enterprise grade SAN, however the speed of development from Intel and Samsung will mean the $/GB will drop steeply and disk sizes will increase exponentially (like what we've with SATA in the past 5 years).
Correction: a single component failing should not stop your service, if you have done your job right (either in designing and building, or in finding a vendor to provide the service). But having a single component failing can and still does ruin somebody's day on a regular basis.
At my company, we have gradually been moving away from spinning disks in favor of SSDs. My company does a lot of R&D work, so we have a lot of people doing CAD, simulation, number crunching, etc. For those users, our IT department hasn't built a machine with spinning media in over two years: the performance boost from SSD is outstanding, and the local storage needs are pretty modest. On the back end, our backup solution (daily incremental backups of everyone's machine, hourly for the network storage) uses a cabinet of HDDs, with a RAID of SSDs that contain the backup database / index. (there are tape backups in the mix, too, with offsite storage, but I forget the details). I'm sure if they could afford to create a 100 TB array of SSDs and do away with the spinning discs entirely, they would.
NSA uses hard disk, bigger capacity, you don't store 2GB/yr of data per person in SSD's. They might have some SSD cache, but more likely it's RAM cache, since they have they're own 150 Megawatt power station, its easier to hold things in RAM during data mining and ensure the power will stay on.
Disks for the bulk of the people, RAM cache for the influencers in the graph (people who originate ideas, are tagged by the 15000 cyber staff as potential targets).
If General Keith Alexander took off the limits on surveillance so it could be applied to the USA, I have no doubt he also took off the limits on propaganda too so we could get our share of NSA propaganda on Slashdot.
14000 Cyber soldiers, mean we have plenty here on Slashdot, a lot with mod points.
http://www.wired.com/threatlevel/2013/06/general-keith-alexander-cyberwar/all/
"Alexander’s agency has recruited thousands of computer experts, hackers, and engineering PhDs to expand US offensive capabilities in the digital realm. The Pentagon has requested $4.7 billion for “cyberspace operations,” even as the budget of the CIA and other intelligence agencies could fall by $4.4 billion."
"The forces under his command were now truly formidable—his untold thousands of NSA spies, as well as 14,000 incoming Cyber Command personnel, including Navy, Army, and Air Force troops. Helping Alexander organize and dominate this new arena would be his fellow plebes from West Point’s class of 1974: David Petraeus, the CIA director; and Martin Dempsey, chair of the Joint Chiefs of Staff."
"In May, work began on a $3.2 billion facility housed at Fort Meade in Maryland. Known as Site M, the 227-acre complex includes its own 150-megawatt power substation, 14 administrative buildings, 10 parking garages, and chiller and boiler plants. The server building will have 90,000 square feet of raised floor—handy for supercomputers—yet hold only 50 people. Meanwhile, the 531,000-square-foot operations center will house more than 1,300 people. In all, the buildings will have a footprint of 1.8 million square feet. Even more ambitious plans, known as Phase II and III, are on the drawing board. Stretching over the next 16 years, they would quadruple the footprint to 5.8 million square feet, enough for nearly 60 buildings and 40 parking garages, costing $5.2 billion and accommodating 11,000 more cyberwarriors."
I recently was given the task of upgrading my development machine. We're a small company but management is happy to spend money on hardware if we need it.
I decided I'd prefer an SSD and yet when I looked at the big suppliers of office machines - Dell, HP, etc. none of them even offered SSD's as an option. SSD's only came into it when you started looking at the really high-end, £2,000+ workstations but there's no reason why this should be the case.
In the end, I just custom built the machine as it was the only way to get the hardware I needed without having to fork out for workstation graphics (which I didn't need).
+1 IDisagreeSoHeMustBeATrollOrAnAstroturferOrAShill
We have spent over a million on a SSD setup for a single host. I'm talking about IBM's V7000's behind IBM SVC. And this is considered Mid-Range, cheaper stuff.
Thing is, when you move from several hundred spinning disks to several dozen SSD's, we it's just not that impressive.
It does the job sure, but it's not Eureka!
The main concern I see (from a storage *network* perspective) is the cost of flash along with write issues (limited write cycles and speed deteriorating on an erase/write), but there are a few companies attempting to lessen the impact of those problems. PureStorage and EMC's XtremeIO attempt to increase efficiency and minimize write-wearing by using inline deduplication with a layer of cache at the controller level. One or both of them also write the data down in a RAID-3 type fashion. Save an entire stripe in cache, then lay it down on the flash to ensure you're not having to go back and fill in spaces, possibly doing erase/writes. Now...I haven't gotten to fully test one, but I'd like to see what happens when you start deleting and fragmenting data. That one should be interesting.
From an internal storage perspective, FusionIO is magnificent (there are others, but I'm not too concerned), though there are a few sticking points that mess with me. Transient data database I've seen thrown on it blaze through like nothing I've ever seen, but if you want any data replicated (to my knowledge, correct me if I'm wrong) you have to do operating system or database level replication. You have a blisteringly fast storage device that finally brings the storage up to speed with the new CPU, and, if you replicate, you start eating into the CPU cycles you'll need to push that storage to the limit.
I was actually curious about the power consumption so I went poking around and found this(Sorry I couldn't find the original article. The power consumption is markedly different....not sure it's enough to COMPLETELY offset the cost, but certainly makes it easier to swallow.
Monstar L
i see you haven't met my teenage daughter.
SSD's might not be used as primary storage, yet. The cost of using a lot of SSD's in a SAN is still too high. However, that doesn't mean that SSD technology is not being used. Many systems started using SSD's as Read/Write caches or highspeed buffers, etc. The PCIe SSD cards are popular in highend servers. This is one way that Oracle manages to blow away the competition when benchmarks are compared. They put a PCIe SSD cards into their servers and use them to run their enterprise database at lightning speeds! ZFS can use SSD's as Read/Write caches although you had better battery backup the Write cache!.
Depending on a particular solution, a limited number of SSD's in a smaller NAS/iSCSI RAID setup can make sense for something that needs some extra OOMF! But I don't yet see large scale replacement of traditional spinning rust drives with SSD's yet. In many cases, SSD's only make sense for highly active arrays where reads and writes are very heavy. Lots of storage sits idle and isn't being pounded that hard.
Yes and no. It's kind of a stupid question, because "business-sized" is undefined. Different people are doing different things. Some of them need big storage, some need less.
I trust SSDs more than physical spindles any day.
Based on what evidence? Where is your data? Faster != More reliable. Spindle based hard drives are (usually) quite reliable and there is plenty of real world usage data documenting exactly how reliable they are. Companies with big data centers like Google have extremely detailed reliability performance figures. SSDs have a lot of advantages but they only recently have started receiving wide distribution and to date they have poor market penetration in data centers where it is easiest to measure their reliability in the real world. Manufacturers estimates of reliability don't mean much in the real world since they have an obvious conflict of interest.
I have little doubt that SSDs will over time replace spinning platters in most places but claims regarding their reliability in relation to spinning platters is somewhat premature, especially in a data center environment. I wouldn't be the least bit shocked to find out they were more reliable (having no moving parts helps a lot) but just because they should be doesn't mean they will be.
Two years on and this is still relevant: The Hot/Crazy Solid State Drive Scale.
I love SSD's in servers and they don't burn me because I always expect them to fail. Sure, one MLC SSD is fine for a ZFS L2ARC, because if it fails reads just slow down, but for a ZFS ZIL, that gets a mirror of SLC drives, because a failure is going to be catastrophic.
If I'm using Facebook's FlashCache, two drives get mirrored by linux md and treated as a cache device and smartd lets me know when one of them goes TU. Another advantage here is linux md is hot-replaceable while pure FlashCache isn't. I just got linux 3.9 on my first server this weekend (thanks, ElRepo) and haven't yet tried Redhat's dm-cache, but the same logic ought to apply; it's only the abstraction and syntax that differs.
Yeah, there's a write penalty with mirrors, but SSD mirror writes are way faster than the best pure spinning-rust RAID (no rotational latency), so it's way better than other options.
Clustering can help too. The other strategy is to make nothing redundant except for a massive cluster of servers. I haven't benchmarked the two strategies (I tend to work with smaller clusters in small businesses) but tech time isn't free either. I suspect at Megalocorp scales where there are several people whose job it is to replace failing disks all day (this is a real thing), going redundant on the compute node scale is a better option. My systems tend to be remote in far-away data centers and nobody wants to have to touch them more than every few months.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
But the real *issue* here is being able to actually go though the data looking for information. Storage of this much data has been a fairly easy problem to solve if you have money, finding a way to organize and search though huge data sets to give timely results is not so easy even if you have money.
Buying spindles and connecting them in huge RAID arrays is well understood. You just build what size you need and dump your data onto it. Yea, you will have to battle OS size limits on partitions and files, but that's not too bad or very expensive. As you point out, getting the hardware isn't that expensive, even at the apparent scale involved here. Buying enough power to turn it all on and keep it cool shouldn't be an issue either, but you need to include that in the $4 Billion budget. In short, if you have money, getting the hardware off the shelf is easy. Software for this is NOT off the shelf.
The REAL money is going towards the software systems that mine the information being collected. There is no system configuration running MySQL that's going to be able to support ongoing data collection (inserts) and any kind of meaningful query results on a petabyte sized data base. I'm guessing that half their budget goes to research and development of software and systems used to collect, store and mine the data. I'm also guessing that they spend roughly 40% of their hardware budget on processing, 40% on storage and 20% on maintenance and operating costs. This puts their hardware budget ($4 Billion * 50%) * 40% or about $1 Billion, give or take.
This means that your 2 Gig turns into about 1/4th that, not accounting for the space being thrown away because it is obsolete. I'm guessing there really isn't that much being kept around on folks who are not interesting, however that is defined.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
Well, if that single component is the main circuit breaker to your server building, I think it will still take it down.
Yeah. When we're talking RAM, we are talking modern interfaces, such as DDR (now DDR2 or DDR3), whereas NAND flash, which is what is used here, uses page mode read & writes. Not to mention that internal writes, which are there on flash but not on RAM, would automatically slow down the process, even if the same interface were used (compare SRAM and NOR flash, as a comparison point).
I think what's contributing to the confusion is SSDs being available not just in SATA interfaces, but now, in PCI-X interfaces as well. SATA doesn't enable SSDs to be much faster than HDDs, but PCI-X does. So having SSDs on multiple PCI-X slots makes faster storage available, and reduces the need for more DRAM, since the latencies in cases that data is not in RAM, would be much lower.
This guy basically nails it. SSD will slowly move it's way from "Tier 0" out, and eventually in 20 or 30 years even our near-line will be SSD, quite possibly.
As I read through this post I kept asking myself "How old is this? Am I in the archives?"
We've been running SSD drives for years. There is no way our VDI infrastructure could scale as it does using old fashioned platter based disks. Our VDI SAN has a shelf of enterprise SSD, and our server SAN has automated shifting of blocks of data from platter based disks to SSD to the enormous amount of RAM located in the SAN.
The answer is "Yes, SSD is ready for business-sized workloads. It has been for years."
Flash writes are a lot slower than SRAM or DRAM writes, but the reads are still very fast, and both of them are a lot faster than rotating mechanical disk drives. Also, the price/performance means that you may be able to afford putting a lot more of it into your machine.
Depending on your application, you may be getting a big performance win by moving from disk drive to some kind of RAM, and it doesn't matter if it's flash or DRAM because the important thing was eliminating rotation and seek latency. Or you might get better performance because you can put a terabyte of SSD onto your machine, while you don't have room (or budget) for that much flash.
Back in the 80s, the price and capacity of RAM for Vaxen improved to the point that we could finally upgrade our machine to 16 MB and still fit in two cabinets. The run-time for our 12-MB simulation application suddenly went from a week down to an hour per run, and I was able to stop chasing mysterious virtual memory flakiness. The upgrade probably cost us about half a person-year's salary, and it would have probably made sense to get it a year earlier, but capital expenses and salaries came from different budgets, and big bureaucratic companies are that good at cost accounting.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
(insert "series of tubes" joke here.)
You don't just care about the acceleration speed; that 1995 Civic had pretty good handling, and might be more use on the freeway than a muscle car that's made to go really fast in a straight line. Or maybe what you need is a truck, because carrying everything in one trip is a lot faster than 50 slightly faster trips.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Operating systems have had good support for 2-tier storage for a while - disk drives cached in RAM. But Flash/SSD really offers an intermediate performance level, and I haven't seen much from either Linux or Windows to take advantage of it without lots of customization or niche applications (such as Readyboost, or mounting /usr/share on a flash stick or whatever.) Has anybody seen anything interesting happening to take advantage of flash?
Enterprise storage solutions do a bit better job in hiding SSD or RAM caching into big expensive SANs, and for a long time they've had solutions for tape backup, etc. But what I've really been looking for is support for a moderate amount of fast disk drive and a large amount of slow bulk storage (e.g. use the expensive SAS drives for the database but archive the logfiles onto consumer-priced slow storage.)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
It depends how many disks it replaces to get the same throughput, and at other times an individual machine will use more power with an SSD because it will spend less time in an IO-WAIT state. In general SSDs reduce the amount of equipment in total needed on most loads because of higher potential processor usage.
I see a lot of people referring to disk arrays as "SANs", which is incorrect. A SAN (storage area network) refers to the network itself, not the array(s). Calling an array a SAN is like calling a networked PC a LAN!
Sorry, just a pet peeve of mine...
A string walks in to a bar. Bartender says 'No Strings!' ... goes out and gets all twisty in the lane, goes back in. Bartender says 'I said no strings!'. He says 'I'm a freyed knot'!