Slashdot Mirror
Data Storage Capacity Mostly Wasted In Data Center
Lucas123 writes "Even after the introduction of technologies such as thin provisioning, capacity reclamation and storage monitoring and reporting software, 60% to 70% of data capacity remains unused in data centers due to over provisioning for applications and misconfiguring data storage systems. While the price of storage resource management software can be high, the cost of wasted storage is even higher with 100TB equalling $1 million when human resources, floor space, and electricity is figured in. 'It's a bit of a paradox. Users don't seem to be willing to spend the money to see what they have,' said Andrew Reichman, an analyst at Forrester Research."
I don't know about your data center, but ours keeps drives well below full capacity intentionally.
The more disk arms you spread the operations over, the faster the operations get, and smaller drives are often more expensive than larger ones.
Plus, drives that are running close to full can't manage fragmentation nearly as well.
I didn't know that I've got $25000 dollars worth of storage at home :-)
The Tao of math: The numbers you can count are not the real numbers.
Likelihood that I get fired because something important runs out of storage and falls over(and, naturally, it'll be most likely to run out of storage under heavy use, which is when we most need it up...): Relatively high...
Likelihood that I get fired because I buy a few hundred gigs too much, that sit in a dusty corner somewhere, barely even noticed except in passing because there is nobody with a clear handle on the overall picture(and, if there his, he is looking at things from the sort of bird's eye view where a few hundred gigs looks like a speck on the map): Relatively insignificant...
for a storage monitoring system.
It's so easy to over-provision. Hardware is cheap and if you don't ask for more than you think you need, you may end up (especially after the app becomes popular, gasp!) needing more than you thought at first.
It's like two kids fighting over a pie. Mom comes in, and kid #1 says "I think we should split it equally". Kid #2 says "I want it all". Mom listens to both sides and the kid who wanted his fair share only gets one quarter of the pie, while the kid who wanted it all gets three quarters. That's why you have to ask for more than you fairly need. It happens not just at the hardware purchase end but all the way up the pole. And you better spend the money you asked for or you're gonna lose it, too.
Who cares if you leave disks 10% full? To get rid of the minimum of 2 disks per server you need to boot from SAN, and disk space in the SAN is often 10x the cost of standard SAS disks. Especially if the server could make do with the two built-in disks and save the cost of an FC card + FC switch port.
I/O's per second on the other hand cost real money, so it is a waste to leave 15k and SSD disks idle. A quarter full does not matter if they are I/O saturated; the rest of the capacity is just wasted, but again you often cannot buy a disk a quarter of the size with the same I/O's per second.
Finally! A year of moderation! Ready for 2019?
Having too much storage is an easy problem. Sure it cost a bit more, but not prohibitively so or you'd never have gotten approval to spend the money. Not having enough storage, OTOH, is a hard problem. Running out of space in the middle of a job means a crashed job and downtime to add more storage. That probably just cost more than having too much would've, and then you pile the political problems on top of that. So common sense says you don't provision for the storage you're going to normally need, you provision for the maximum storage you expect to need at any time plus a bit of padding just in case.
AT&T discovered this back in the days when telephone operators actually got a lot of work. They found that phone calls tend to come in in clumps, they weren't evenly distributed, so when they staffed for the average call rate they ended up failing to meet their answer times on a very large fraction of their calls. They had to change to staffing for the peak number of simultaneous calls, and accept the idle operators as a cost of being able to meet those peaks.
This is the CYA approach, and I don't see it getting any better. When configuring a server, it's usually better to pay the marginally higher cost for 3-4x as much disk space as you think you'll need, rather than risk the possibility of returning to your boss asking to buy MORE space later.
"People who think they know everything are very annoying to those of us who do."-Mark Twain
Interesting. Was the culprit all cad files out of the new rev?
Yes, for the most part. Because of a bad config, they were going from drawings around 1-10MB to drawings over 100MB. That's what happens when you get management to take the IT department out of the software management and configuration equation. We were, of course, still left to sweep up the pieces.
time to go and buy up all kinds of expensive software to tell us something or other
it's almost like the DR consultants who say we need to spend a fortune on a DR site in case a nuclear bomb goes off and we need to run the business from 100 miles away. i'll be 2000 miles away living with mom again in the middle of no where and making sure my family is safe. not going to some DR site that is going to close because half of NYC is going to go bankrupt in the depression after a WMD attack
This isn't like an ISP overbooking a line and hoping that everyone doesn't decide to download a movie at the same time. If a hosting service says your account can have 10GB of storage, contractually they need to make sure 10GB of storage exists.
Even though most accounts don't need it.
One client of mine dramatically over-provisioned his database server. But then again, he expects at some point to break past his current customer plateau and hit the big time. Will he do so? Who can say?
It may be a bit wasteful to over-provision a server, but I can guarantee you that continually ripping out "just big enough" servers and installing larger ones is even more wasteful.
Your pick.
Any sect, cult, or religion will legislate its creed into law if it acquires the political power to do so.
"It's a bit of a paradox. Users don't seem to be willing to spend the money to see what they have,"
I think he meant users don't seem willing to spend the money to MANAGE what they have.
As many have pointed out, you need 'excess' capacity to avoid failing for unusual or unexpected processes. How often has the DBA team asked for a copy of a database? And when that file is a substantial portion of storage on a volume, woopsie, out of space messages can happen. Of course they should be copying it to a non-production volume. Mistakes happen. Having a spare TB of space means never having to say 'you're sorry'.
Aside from the obvious problems of keeping volumes too low on free space, there was a time when you could recover deleted files. Too little free space pretty much guarantees you won't be recovering deleted files much older than, sometimes, 15 minutes ago. In the old days, NetWare servers would let you recover anything not overwritten. I saved users from file deletions over the span of YEARS, in those halcyon days when storage became relatively cheap and a small office server could never fill a 120MB array. Those days are gone, but without free space, recovery is futile, even over the span of a week. Windows servers, of course, present greater challenges.
'Online' backups rely on delta files or some other scheme that involves either duplicating a file so it can be written intact, or saving changes so they can be rolled in after the process. More free space here means you actually get the backup to complete. Not wasted space at all.
Many of the SANs I've had the pleasure of working with had largely poor management implementations. Trying to manage dynamic volumes and overcommits had to wait for Microsoft to get its act together. Linux had a small lead in this, but unless your SAN lets you do automatic allocation and volume expansion, you might as well instrument the server and use SNMP to warn you of volume space, and be prepared for the nighttime alerts. Does your SAN allow you to let it increase volume space based on low free space, and then reclaim it later when the free space exceeds threshold? Do you get this for less than six figures? Seven? I don't know, I've been blessed with not having to do SAN management for about 5 years. I sleep much better, thanks.
Free space is precisely like empty parking lots. When business picks up, the lot is full. This is good.
deleting the extra space after periods so i can stay relevant, yeah.
There are two numbers that matter for storage systems. One is the raw number of gigabytes that can be stored. The other is the number of IO's that can be performed in a second. The first limits the size of the collected data. The second limits how many new transactions can be processed per time period. That, in turn, determines how many pennies we can accept from our customers during a busy hour.
We size our systems to hit performance targets that are set in terms of transactions per second, not just gigabytes. Using round numbers, if a disk model can do 1000 IO/second, and we need 10,000 IO/second for a particular table, then we need at least 10 disks for that table (not counting mirrors). We often use the smallest disks we can buy, because we don't need the extra gigs. If the data volume doesn't ever fill up the gigabyte capacity of the disks, that's ok. Whenever the system uses all of the available IO's-per-second, we think about adding more disks.
Occasionally a new SA doesn't understand this, sees a bunch of "empty" space in a subsystem, and configures something to use that space. When that happens, we then have to scramble, as the problem is not usually discovered until the next busy day.
I'm not sure if you're trolling or not, but if you're serious did you happen to manage the storage for Microsoft's Sidekick servers?
A couple things wrong with your assumptions:
1) 1TB drives might be great for storing your goat porn collection, but on a server with actual load, how many of those drives do you need to get adequate IOPS? Also exactly 100 of them means no RAID, but that's OK because drives from Newegg never fail so your 100TB of data should be fine.
2) You seem to have left controllers out of your list. Anyone who's ever had a RAID controller start barfing garbage all over a LUN, or take out a second drive after a drive failure will tell you the controller is the really critical bit (and is usually a single point of failure in systems with DAS.)
3) Where's your backup hardware? Where's space for snapshots? Where's space for replication?
4) Ever time a RAID5 rebuild on say a 9 drive LUN with 1TB SATA disks?
Storage is expensive because the data on it has value and making sure that data is available and isn't lost or corrupted costs money. Cheap storage solutions don't end up that way when the drives have to go to OnTrack for recovery and the company's down for a week, or valuable data is lost.
Fsck the millennium, we want it now.
Millennium Crisis Line: 0890 900 2000 [calls cost 50p/min]
Most SAN administrators wouldn't be caught dead using your $130 1TB drives. Rerunning your calculations with 15K 450GB SAS drives (around $300 bucks), and you're spending quite a bit more: 228 drives will give you 100TB, sure, but we'd want some redundancy . . . say RAID 5 (not the best approach for SAN design, but let's keep it simple) which pushes the drive count up to 304 with a total cost of $91,200, just for disks. To get a real, enterprise enclosure (or rather, cluster of enclosures considering the drive count) that offers things like FiberChannel, 10Gb iSCSI, or InfiniBand uplinks, and features such as SAN to SAN replication, bit deduplification, and other enterprise-level utilities/features, I'd say you're looking at $500,000 (ballpark guess) just to have something to stick the drives into. We're at ~$600,000 without even taking into account the physical costs of operation, datacenter architecture, or labor costs to maintain such a SAN.
Suddenly, that $1 million isn't so far fetched, eh?
You lost something along the way. When you are doing RAID 5 on an enterprise array, you are likely using 5+1 sets. Your 304 drives does not take into account losing 2 drive capacity every 6 drives. You can get away with global hot sparing, but that doesn't cover your ass as much. You would need 342 drives.
We do use thin provisioning, and virtualization in general, but I agree that there is benefit to keeping utilization low. We try to keep more space than we could possibly need both because it can sometimes surprise you by growing quickly and because the drives are faster if the data is spread across multiple drives. Also SSD drives sometimes live longer if not fully utilized, because they can distribute the wear and tear, so we usually leave 20% unformatted.
Downtime and slow systems are much more expensive than wasted drive space.
At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
i've seen worse. At my company they moved the CAD software management to drafters and then they broke up the drafting department and just assigned each drafter to a team. I am an engineer and i sit near the IT department. I feel sorry for the poor buggers, now not only do they have to run around like headless chooks. But so do the CAD drafters because before the load level was done by a head drafter allocating work. now its managers running around asking other managers can they "borrow" there drafter, and we have different people running different versions and to sum it up its hell to watch.
And the only reason they implemented such a scheme was that accounting told them it would save money... So instead of having 8 drafter for the whole company we now have 12 (one for each project). Sometimes the world doesn't work with just numbers!