Slashdot Mirror
Data Storage Capacity Mostly Wasted In Data Center
Lucas123 writes "Even after the introduction of technologies such as thin provisioning, capacity reclamation and storage monitoring and reporting software, 60% to 70% of data capacity remains unused in data centers due to over provisioning for applications and misconfiguring data storage systems. While the price of storage resource management software can be high, the cost of wasted storage is even higher with 100TB equalling $1 million when human resources, floor space, and electricity is figured in. 'It's a bit of a paradox. Users don't seem to be willing to spend the money to see what they have,' said Andrew Reichman, an analyst at Forrester Research."
I don't know about your data center, but ours keeps drives well below full capacity intentionally.
The more disk arms you spread the operations over, the faster the operations get, and smaller drives are often more expensive than larger ones.
Plus, drives that are running close to full can't manage fragmentation nearly as well.
The cost of too much storage isn't bad.
Of course - you may say that it's necessary to delete old data, but in some cases you can't know which old data that may be needed again.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
I want more toys.
I didn't know that I've got $25000 dollars worth of storage at home :-)
The Tao of math: The numbers you can count are not the real numbers.
Likelihood that I get fired because something important runs out of storage and falls over(and, naturally, it'll be most likely to run out of storage under heavy use, which is when we most need it up...): Relatively high...
Likelihood that I get fired because I buy a few hundred gigs too much, that sit in a dusty corner somewhere, barely even noticed except in passing because there is nobody with a clear handle on the overall picture(and, if there his, he is looking at things from the sort of bird's eye view where a few hundred gigs looks like a speck on the map): Relatively insignificant...
for a storage monitoring system.
The story would be generating much gnashing of teeth about the evil corporations and the corner cutting that was bringing down our pink unicorns.
Can win for losing around here.
It's so easy to over-provision. Hardware is cheap and if you don't ask for more than you think you need, you may end up (especially after the app becomes popular, gasp!) needing more than you thought at first.
It's like two kids fighting over a pie. Mom comes in, and kid #1 says "I think we should split it equally". Kid #2 says "I want it all". Mom listens to both sides and the kid who wanted his fair share only gets one quarter of the pie, while the kid who wanted it all gets three quarters. That's why you have to ask for more than you fairly need. It happens not just at the hardware purchase end but all the way up the pole. And you better spend the money you asked for or you're gonna lose it, too.
Who cares if you leave disks 10% full? To get rid of the minimum of 2 disks per server you need to boot from SAN, and disk space in the SAN is often 10x the cost of standard SAS disks. Especially if the server could make do with the two built-in disks and save the cost of an FC card + FC switch port.
I/O's per second on the other hand cost real money, so it is a waste to leave 15k and SSD disks idle. A quarter full does not matter if they are I/O saturated; the rest of the capacity is just wasted, but again you often cannot buy a disk a quarter of the size with the same I/O's per second.
Finally! A year of moderation! Ready for 2019?
Interesting. Was the culprit all cad files out of the new rev?
Having too much storage is an easy problem. Sure it cost a bit more, but not prohibitively so or you'd never have gotten approval to spend the money. Not having enough storage, OTOH, is a hard problem. Running out of space in the middle of a job means a crashed job and downtime to add more storage. That probably just cost more than having too much would've, and then you pile the political problems on top of that. So common sense says you don't provision for the storage you're going to normally need, you provision for the maximum storage you expect to need at any time plus a bit of padding just in case.
AT&T discovered this back in the days when telephone operators actually got a lot of work. They found that phone calls tend to come in in clumps, they weren't evenly distributed, so when they staffed for the average call rate they ended up failing to meet their answer times on a very large fraction of their calls. They had to change to staffing for the peak number of simultaneous calls, and accept the idle operators as a cost of being able to meet those peaks.
If you RTFA (and admittedly, this is not very clear), the article tries to make the point that you don't need all of this storage capacity to be live. However, you've got a bunch of storage pools or machines just running idling as opposed to actually doing something. What the article is trying to say is that using provisioning tools that will spin up storage pools or servers as they are needed (as capacity increases) is a much better solution to just leaving them running. Obviously peak load will cause issues, but you configure your provisioning tools to be smarter to start bringing up capacity at lighter loads or specific times of day. The point still stands that most data centers just have idling machines that could just as easily be shut off most of the time and automatically brought up when needed, it's just that most do not use these tools despite the savings in electricity, wear, and cooling costs.
The article confounds the issue by starting to talk about the lack of monitoring tools that leads to overprovisioning, and ends with a discussion as to how to make the storage problem more efficient (thin provisioning). Thing is, thin provisioning only works when you have the extra capacity, but it's not live until you need it. You still need to overprovision, but you won't be running all those resources idle at once just in case.
2 146GB drives from HP are less than $500 for the SAS drives. you can put the same storage on an EMC SAN and provision less for the system drive for a Windows server but by the time you pay their crack dealer prices for hard drives along with the drives for the BCV volumes and pay for the fiber switches and g-bics and HBA's and everything else it's cheaper to waste space on regular hard drives
This is the CYA approach, and I don't see it getting any better. When configuring a server, it's usually better to pay the marginally higher cost for 3-4x as much disk space as you think you'll need, rather than risk the possibility of returning to your boss asking to buy MORE space later.
"People who think they know everything are very annoying to those of us who do."-Mark Twain
OK, bare 1TB enterprise class drives cost about $130 at Newegg retail. (half that price if you go for standard grade disks)
A hundred such disk drfives will set you back $13,000.
Figure another $10,000 for mounting, power supplies, connectors, and other obvious hardware.
Another $2,000 for four racks.
Floorspace? Racking them loosely gives you 25 per rack or 4 racks. Each rack is about 10 square feet, or 40 square feet ... roughly the same as the power cost ... another $1100 per year.
At $10/square foot, that's maybe $400 or $500 a month or around $5,000 per year
Electricity? 100 drives at 8 watts per drive yields full time load of 800 watts;
at a nominal $0.15 per KWhr, that's around $1100 per year in electric bills.
Air Conditioning
Replacement at 2 percent failure rate is perhaps $200 per year.
Human costs? The cost of labor to support a 50 TB disk farm can't be much different from that of a 100TB farm.
Indeed, it's probably less labor (and software) intensive to have a system with great overcapacity than one that needs squeezing.
In either case, at most, a 100TB disk farm might need 2 full time staffers. Generously, that's $150,000 per year.
So hardware costs of a 100TB system is around $25,000.
And annual operating costs of around $3,000 per year.
Labor costs of $150,000 per year.
Where do they get the $1,000,000 per year?
Shitloads of unused disk space is what I *want*.
time to go and buy up all kinds of expensive software to tell us something or other
it's almost like the DR consultants who say we need to spend a fortune on a DR site in case a nuclear bomb goes off and we need to run the business from 100 miles away. i'll be 2000 miles away living with mom again in the middle of no where and making sure my family is safe. not going to some DR site that is going to close because half of NYC is going to go bankrupt in the depression after a WMD attack
This isn't like an ISP overbooking a line and hoping that everyone doesn't decide to download a movie at the same time. If a hosting service says your account can have 10GB of storage, contractually they need to make sure 10GB of storage exists.
Even though most accounts don't need it.
One client of mine dramatically over-provisioned his database server. But then again, he expects at some point to break past his current customer plateau and hit the big time. Will he do so? Who can say?
It may be a bit wasteful to over-provision a server, but I can guarantee you that continually ripping out "just big enough" servers and installing larger ones is even more wasteful.
Your pick.
Any sect, cult, or religion will legislate its creed into law if it acquires the political power to do so.
This is one of the arguments that's made for using a SAN. Consolidate to make better use of the disk space. Smaller footprint, less power, etc.
Yes Francis, the world has gone crazy.
Thin provisioning doesn't fix this problem. At least not today.
The only way thin provisioning fixes this problem is if you over-commit the thin pool. That's all well and good, but currently, any given storage chunk that is allocated to a server is stuck being allocated to that server. So, if I were a server admin who found out he'd been given thin LUNs in an over-commited pool, I know that if my neighboring admins don't keep track of their storage use, then my server could wind up crashing because they took up all the storage. So instead, I'm going to write a script first thing when I get the storage to write a text file clear across the drive. There. Now my disk is fully provisioned, and my neighbors can use all the pool they want, it won't affect me. 'course, not everyone can do that, or the pool will fill up lickety split.
Now, someday, the servers will be smart enough to tell the storage array when they're done with a chunk of storage. At which point, that part of the pool can be freed up. When that happens (and it will, but it's going to take some time), thin pools will be ideal. Everyone will have all the storage they need almost all of the time.
However, that day isn't here yet. In the mean time, there are interesting performance reasons to use thin provisioning, but not space-related ones.
70% used space with the 100TB mentioned in the article, leaving us with 70TB.
Think of how much porn 70TB is!
So ... 100 TB / 1 Million ==> 1 TB / $10,000.
A 1 TB drive is 60-100 dollars. .12 cents per KW/h, that's 63.12 per year.
The KW/h required to run a 60 watt drive 24/7 = 60/1000 KW x 24 hours x 365.25 days = 526 KW/h.
At
Even if we double or triple the hardware costs, they will only make up a few percentages of the 10 grand per TB cited here.
The labor to maintain 100 or 200 or 400 drives is going to be relatively constant. In fact, with a little more reasonable monitoring software (just reporting drive failures in a raid system, so the labor just has to pull bad drives and replace with good drives), I don't think the capacity of a data center is all that related to the labor costs.
End result, it is just cheaper and easier to throw hardware at problems to reduce labor costs than to pay for expensive software to monitor capacity and be more efficient in the use of capacity.
Instead of a medium number of large systems, I wonder whether it would make more sense to have a larger number of mini-itx type units that could be:
- easily replaced
- put in stand-by when no access - smart load balancer would decide when to wake up sleeping units.
- simplified cooling?
It would also be nice for a universal back-plane design to support plugging in boards from any company, with minimal or zero cabling.
Jumpstart the tartan drive.
Billions of dollars are also wasted every year in the manufacturing and transporting of fire extinguishers, 99% of which will probably never be used.
explosion(s) in the straits of hormuz. anybody who's read either the bible, or playboy should be able to cipher out what that means. there was even a sci-fi book/movie about it.
"It's a bit of a paradox. Users don't seem to be willing to spend the money to see what they have,"
I think he meant users don't seem willing to spend the money to MANAGE what they have.
As many have pointed out, you need 'excess' capacity to avoid failing for unusual or unexpected processes. How often has the DBA team asked for a copy of a database? And when that file is a substantial portion of storage on a volume, woopsie, out of space messages can happen. Of course they should be copying it to a non-production volume. Mistakes happen. Having a spare TB of space means never having to say 'you're sorry'.
Aside from the obvious problems of keeping volumes too low on free space, there was a time when you could recover deleted files. Too little free space pretty much guarantees you won't be recovering deleted files much older than, sometimes, 15 minutes ago. In the old days, NetWare servers would let you recover anything not overwritten. I saved users from file deletions over the span of YEARS, in those halcyon days when storage became relatively cheap and a small office server could never fill a 120MB array. Those days are gone, but without free space, recovery is futile, even over the span of a week. Windows servers, of course, present greater challenges.
'Online' backups rely on delta files or some other scheme that involves either duplicating a file so it can be written intact, or saving changes so they can be rolled in after the process. More free space here means you actually get the backup to complete. Not wasted space at all.
Many of the SANs I've had the pleasure of working with had largely poor management implementations. Trying to manage dynamic volumes and overcommits had to wait for Microsoft to get its act together. Linux had a small lead in this, but unless your SAN lets you do automatic allocation and volume expansion, you might as well instrument the server and use SNMP to warn you of volume space, and be prepared for the nighttime alerts. Does your SAN allow you to let it increase volume space based on low free space, and then reclaim it later when the free space exceeds threshold? Do you get this for less than six figures? Seven? I don't know, I've been blessed with not having to do SAN management for about 5 years. I sleep much better, thanks.
Free space is precisely like empty parking lots. When business picks up, the lot is full. This is good.
deleting the extra space after periods so i can stay relevant, yeah.
What is the cost if you have 1% of shortage on your capacity? I am sure it will be more then what you pay for over capacity.
Don't fight for your country, if your country does not fight for you.
What trout. I suspect a large amount of this 'wastage' is due to the fact that the smallest HD's available are into the 100's of GB *.
Many dedicated server users do not waste the space but simply never needed it in the first place. Applications that need a dedicated server do not necessary need the storage that comes with it.
* Currently 160GB on an entry level Dell Server
You forgot to pay the executive.
Unlike in the movie "This is Spinal Tap" there is not an 11 on the volume control for storage capacity in a data center. We will not see proud proclamations from boards of directors "today we are running our data storage at 115% of capacity!"
Having been in the predicament many times of frantically trying to ration out disk storage space for some critical application at 3 AM Sunday morning I think that running data centers at 80-90% is being conservative and may save your ass the next time you cannot get into your data center due to some sort of natural disaster like a hurricane (remember the data center in New Orleans a few years ago?)
Storage space does cost money, when we are looking at terabytes (petabytes anyone?) of storage there does need to be some cost factor calculations. In the telco world we do a similar exercise with Erlang calculations and blocking probability for data circuits. I would rather that the cut-off point between enough or too much storage capacity be made by well informed engineers rather than some clueless MBA looking for a feather in their hat.
Tisha Hayes
100TB for a million dollars is about right when you start looking at enterprise storage solutions, such as Netapp or EMC.
.
Alright, I'm not running a bunch of petabytes in a big datacenter right now, but I've been doing this for a really really long time. Hasn't the rule of thumb always been to have a MINIMUM of 25% capacity free? Everyone has always been much more comfotable with 50% free. This old school rule of thumb applies at any scale, megabyte to petabyte, doesn't it?
$130 is for a TB of fast SATA disks. I just had to price out a 6TB Symmetrix VMax SAN w/ EMC (4TB 15K RPM Fibre Channel, 2TB SSD), and the price was $340,000
Minded, the above cost includes the storage controllers, PDU's, rack cabinets, etc. Additional TB's will run us in the neighborhood of $8K-12K (depending on whether we go 15K RPM or SSD)
Spending a million on 100TB of storage for the enterprise is very easily doable, even if you go with slower 10K RPM Fibre Channel disks.
There are two numbers that matter for storage systems. One is the raw number of gigabytes that can be stored. The other is the number of IO's that can be performed in a second. The first limits the size of the collected data. The second limits how many new transactions can be processed per time period. That, in turn, determines how many pennies we can accept from our customers during a busy hour.
We size our systems to hit performance targets that are set in terms of transactions per second, not just gigabytes. Using round numbers, if a disk model can do 1000 IO/second, and we need 10,000 IO/second for a particular table, then we need at least 10 disks for that table (not counting mirrors). We often use the smallest disks we can buy, because we don't need the extra gigs. If the data volume doesn't ever fill up the gigabyte capacity of the disks, that's ok. Whenever the system uses all of the available IO's-per-second, we think about adding more disks.
Occasionally a new SA doesn't understand this, sees a bunch of "empty" space in a subsystem, and configures something to use that space. When that happens, we then have to scramble, as the problem is not usually discovered until the next busy day.
I'm not sure if you're trolling or not, but if you're serious did you happen to manage the storage for Microsoft's Sidekick servers?
A couple things wrong with your assumptions:
1) 1TB drives might be great for storing your goat porn collection, but on a server with actual load, how many of those drives do you need to get adequate IOPS? Also exactly 100 of them means no RAID, but that's OK because drives from Newegg never fail so your 100TB of data should be fine.
2) You seem to have left controllers out of your list. Anyone who's ever had a RAID controller start barfing garbage all over a LUN, or take out a second drive after a drive failure will tell you the controller is the really critical bit (and is usually a single point of failure in systems with DAS.)
3) Where's your backup hardware? Where's space for snapshots? Where's space for replication?
4) Ever time a RAID5 rebuild on say a 9 drive LUN with 1TB SATA disks?
Storage is expensive because the data on it has value and making sure that data is available and isn't lost or corrupted costs money. Cheap storage solutions don't end up that way when the drives have to go to OnTrack for recovery and the company's down for a week, or valuable data is lost.
Fsck the millennium, we want it now.
Millennium Crisis Line: 0890 900 2000 [calls cost 50p/min]
If you don't mind a bit of DIY there is the Backblaze pod:
http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/
It is a 4U disk server that holds 45 disks. They have made the chassis design available from Protocase.
I suppose you could call this a SAN server but it really is just a bunch of cheap storage. As has been commented earlier, in a data center multiple disks are often bought for performance not space. You gain performance by having multiple sets of heads moving at the same time. RAID cache helps this but does not eliminate it.
Did you factor in how expensive is it to change storage size, and the costs of failing to change it? Also, there is the cost of adding some storage that isn't compatible to the first chunk. The amount you pay for oversized storage normaly isn't even on the same order of magnitude of all of those.
Rethinking email
Aptare's latest version of reporting software, StorageConsole 8, costs about $30,000 to $40,000 for small companies, $75,000 to $80,000 for midsize firms, and just over $250,000 for large enterprises. "Our customers can see a return on the price of the software typically in about six months through better utilization rates and preventing the unnecessary purchase of storage," Clark said. just another industry slashvertisement. nothing to see here that we didnt know about already. please move along.
Good people go to bed earlier.
Most SAN administrators wouldn't be caught dead using your $130 1TB drives. Rerunning your calculations with 15K 450GB SAS drives (around $300 bucks), and you're spending quite a bit more: 228 drives will give you 100TB, sure, but we'd want some redundancy . . . say RAID 5 (not the best approach for SAN design, but let's keep it simple) which pushes the drive count up to 304 with a total cost of $91,200, just for disks. To get a real, enterprise enclosure (or rather, cluster of enclosures considering the drive count) that offers things like FiberChannel, 10Gb iSCSI, or InfiniBand uplinks, and features such as SAN to SAN replication, bit deduplification, and other enterprise-level utilities/features, I'd say you're looking at $500,000 (ballpark guess) just to have something to stick the drives into. We're at ~$600,000 without even taking into account the physical costs of operation, datacenter architecture, or labor costs to maintain such a SAN.
Suddenly, that $1 million isn't so far fetched, eh?
I've got "Precise" in quotes because I'm skeptical that you can ever get really good predictions about the future, or even the present, especially from users. But if you try, it's going to take you a while, and you'll be spending loaded-engineer-salary time and harassed-user time trying to get predictions that the users will tell you are guesses. Meanwhile, you do need to get some disks online, and it's going to take you a while to accumulate tracking data. I'm in that kind of situation now - until there's enough disk and users on the system to get a really good model of users, we won't really know, so we're aiming high.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Flood, earthquake, hurricane (yes, possible even in New York), sink hole, etc.
Are you really going to go primeval when any one of those things happens?
First thing, of course you're going to find out if your family is fine. Assuming so, then what? Not only has their home been destroyed, but your job is gone too, so you'll now be dependant on insurance (notoriously unwilling or unable to pay after disasters) and handouts.
Not that you should be spending billions on off-site data storage and redundant systems, but a large company being completely unable to survive the loss of a single building/office is quite short-sighted, even if it happens to cost some money up-front.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
Unfortunately, my company is sufficiently large and bureaucratic that equipment standards are often made by people who don't know the applications :-) The bureaucrats like SAN arrays because they're blazingly fast, and because they're easy to administer, back up, plan for storage growth, etc. And $8000/TB is really just fine if your idea of "huge" storage is a TB or two.
I've got an application that needs to do a bit of fast-IOPS logging (so the overpriced SAS drives and SAN array are fine), but needs lots of bulk storage that doesn't need blazing fast access, but does need to be on disks as opposed to tape. I probably need 10TB to start with, growing to 20+ as we get more customers.
It's obviously a job for NAS - Network Attached Storage, the kind of stuff Netapp used to sell (presumably still does), which wraps a certain amount of framework around the big cheap drives, so you still get the operational benefits and manageability, at a cost that's maybe twice what the cheap raw drives cost. Lacking that, I'd been thinking about stuffing a server full of 2TB drives, but our IT folks not only don't like that because they don't have the manageability and easy upgrades, but they only allow little 300GB SAS drives (not even the new 600GB), and only support 2.5" drives, not 3.5", because that's more cost-effective or something. Sigh.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
> in case a nuclear bomb goes off
Or even more far-fetched, someone brings in a fan heater from home, forgets to switch it off one evening, some paper blows into the elements and sets on fire, and it burns down the building.
Keeping an off-site backup is not a ridiculous idea in itself. Could the business survive if the office burned down and all servers and data was lost? Maybe if employees are allowed to take data home, most stuff could be pieced back together, but even then it would be a substantial amount of work. But as with TFA, it's not something to spend a massive amount of money on. Where I work, all projects should have a daily backup to a central server (just simple batch script / shell script / version control system), and that has an off-site backup, which as far as I'm aware just means one of the admins swaps the hotswap bays and takes the discs home on a weekly basis. Total cost is about 5 minutes a week to swap the discs; the hotswap hardware itself, and a few extra discs is well under £1000. Everything else is no different from what we'd need to do for regular backups anyway.
Even for my own data, e.g. holiday photos, every so often I make sure to put it all on a removable hard disc and copy it onto my work PC. I'd certainly consider it worthy of a disaster recovery solution, given that it's so very easy to do.
Nice job! I applaud this effort to put wild claims into perspective.
Corporate accounting == magic, for some perverse value of magic. Every organization that touches that datacenter wants to claim some cost or another on their balance sheet, so they can be seen spending money in such an unarguably useful (and permanently reoccurring) way and ensure they get more next year. The money should only get spent once, but money circulates around from one department to another and might actually get "spent" several times before it moves to some other part of the corporate bloodstream. Meanwhile, the datacenter probably has no budget of its own.
This happens more often than not in corporations large enough to set up a 100 TB farm. I'd love to work at that fictional company, because $75k is a pretty nice salary... more than I make right now doing similar work! I don't have that sort of farm where I work, though.
Your biggest datacenter costs will probably come from networking hardware, software licensing and support contracts. Database servers are particularly odious in this respect. $25k per socket for MS SQL Server, for one - if you have two servers with dual CPUs, that's $100k right away just for the database software. Probably more for Oracle. Usually it's some proprietary software dependency that locks you into a specific DB vendor. Then there's the OS licensing. You can spend $2k or more on Windows Server, and adding seats for Terminal Services gets expensive fast. What if you want Exchange, which is a big reason for having lots of spindles? Blackberry? Nice Cisco routers? Telecom equipment and vendor support from the telco? Ka-ching!!! Then there's the applications your datacenter supports. Proprietary software from other vendors will have large support contract bills which can rise to the tune of $1M or more. When you have vendor lock-in because all the organizations who touch your datacenter use a certain brand of product, you're kinda stuck with that bill.
So, if you're running an all-Linux, all-open-source datacenter, there's no licensing/support fees. BUT, good luck selling that idea! Management wants a finger to point and someone at the other end of that finger to say "yes sir, we're sorry, we're fixing it right now and it will be done in the terms of our SLA", and the datacenter employees want that to be someone other than them!
I would look more to the issue of drive performance as the main cause of this. What happens to a spinning platter hard drive, when it has to read data from half-way in the disc, rather than from the outside? Performance drops anywhere from 45 to 70% once you have your hard drives filled half-way. Naturally, in order to keep high performance on real-time critical data, you have to get higher density platters, and only use the outside of them. Unfortunate as it is, I think this is more of a cause for un-used storage space than under-prescribing.
I have read the article (gasp!) and read alot of the comments here that get into the minutia of the I/O's, read writes, CYA, etc. - Bottom Line - If a company is going to invest in any storage solution, if no one has a baseline that allows for properly sizing a storage solution, it all becomes a crap shoot. What I mean when I say planning is - knowing what you currently have (total space (used/free space)), having a growth curve or trend that shows how much actual DATA growth is experienced year over year (40%, 50%, etc), what the requirements of existing systems are (since MOST new Operating Systems and programs do not significantly expand the intial disk usage requirements over time), and putting that together with the thought of an actual DR (ability to recover). So if a storage solution is properly planned you can cost estimate what you need for how long you believe you can live with said equipment (aka - solution needs to last 8 years requires x space with x backup solution), if not then you deserve to be fired. Management and adminisration at that point become less relevent because you know your preared for everything you tried to take into account before the solution even showed up. In addition, most companies have a pretty defined structure to how and where the data lives, and most good netadmins are watching it like a hawk anyway (scripts or some existing monitoring solution). Lastly, the article does not account that some free or wasted space is REQUIRED, as almost all OS's like to have free space available for swap space and/or other hidden system functions and when you do not have adequate free (aka wasted) space those programs and OS's tend to run like crap (no matter the hardware). So from everything I can gather from this article and not taking into account that free space is usually required (disagree with me get your free space to less than 20% on windows and watch the performance go downhill as that free number moves lower) is really a - move along nothing to see here except someone trying to sell me an overpriced solution i dont need or want at this time. Nothing more than some FUD
You lost something along the way. When you are doing RAID 5 on an enterprise array, you are likely using 5+1 sets. Your 304 drives does not take into account losing 2 drive capacity every 6 drives. You can get away with global hot sparing, but that doesn't cover your ass as much. You would need 342 drives.
We do use thin provisioning, and virtualization in general, but I agree that there is benefit to keeping utilization low. We try to keep more space than we could possibly need both because it can sometimes surprise you by growing quickly and because the drives are faster if the data is spread across multiple drives. Also SSD drives sometimes live longer if not fully utilized, because they can distribute the wear and tear, so we usually leave 20% unformatted.
Downtime and slow systems are much more expensive than wasted drive space.
At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
Working for a storage vendor, I would say even your numbers are low. Fat chance getting a 1TB drive for less than $800! Why so much? Beats me. Maybe it's the really nice sleds they are mounted in? They are high grade plastic after all...
Or at least the whole truth, quite often you find that to replicate your 8 TB volume really requires you to buy a SAN with 16 TB capacity on one end and 16 TB on the other with the "unused" space for replication overhead or whatever fancy SAN tricks you want play.
So while you wanted 16 TB of capacity, you actually buy 32+ much of what appears to be uncommitted.
I can't be sure which one. On second thought, make that second one ``stupidity''. I still can't decide which one's really at work.
I attend a daily conference call where I hear case after case where people are waiting for SAN space to be reclaimed so that it can be reassigned to other systems. People are either being told their projects cannot proceed because there's not enough disk space or other projects have been held up while the space they've been allocated is scaled back to allow others to work.
I'm not sure what the storage team or (as I suspect) the clueless architects are doing when someone implementing a new application asks for space on the SAN but the procedure seems to be ``whatever the project says it needs, multiply that by ten (or more)''. During one conference call I heard that the project had been placed on hold while some of the disk space they'd been allocated was reclaimed for use on other projects. When I asked how much they were having to give up, I was told that to migrate an existing database from an environment where they were using less than 1TB of disk space they had been given 20TB on the new hardware environment. (Heh... Did I same some of the disk space? I should have said most.)
CUR ALLOC 20195.....5804M
One of the subtle benefits of the computer revolution is that it gives society the ability to be wasteful. Desktop computers that often aren't at 100% cpu utilization, and (local) networks that rarely see peak usage are also good signs, for the same reason. Hard drive capacities obviously aren't quite there yet, but they're getting closer. This might be yet another sign of the singularity.
We happily manage about 75-90tb total storage across our servers (really, i've forgotten how much exactly is total across all different server types), with less than 1$ million. Way less than 1$ million.
Then again, when you go the expensive route (say EMC), it's certainly expensive. But if senses are stuck together, and only acquires features required, storage doesn't need to be expensive. There's plenty of ways of running servers with storage costing almost as little as consumer storage, when some thought has been put into cost-savings, while not sacrificing redundancy. That being said, expensive storage systems DO have their perks, but only at the enterprise level.
Pulsed Media Seedboxes
We're growing so fast that we can barely keep up with the demand. Maybe I can run a few cross connects into my neighbor's cage and borrow some of their unused space?
You calculated a lot of costs there, but you forgot the most important one:
The cost of your business imploding when a critical app falls over because your one hundred, 7200 RPM SATA spindles will only deliver, on a good day with the wind in their sails, 6,000 IOPS at less than 15 ms per op. Or a bad day when you experience multiple drive failures and your shoes are all squishy due to the piss that ran down your leg because you just noticed that your IOPS just dropped to 1,000/sec, your latency just went up to 100 ms and it's going to take over twenty-four hours to complete just one rebuild.
In the data center a few floors below my desk, we have a dozen database servers that average 6,000 IOPS all day long with peak times requiring 10,000 IOPS for several hours every day. Per server.
That's why it's not a hundred drives. It's a couple thousand. And they're not 1TB+ 7200 RPM SATA. They're 146 GB/300GB/400GB 15,000 RPM FC and SAS. And there's over a hundred GB of NVRAM. And everything is connected via multiple paths through two or more separate fabrics (i.e. at least two of these. Each of which will run you over $100k just to get on the floor. Plus annual support and maintenance costs.)
Regardless, it's not just moving bytes into and out of a hard drive. It's being able to replace failed components... as in every component, things like controllers, front end controllers & ports, back end controllers & ports, batteries, drive back planes, and hard drives. Being able to add physical capacity, new drives, ports, controllers, RAM, etc. It's having off site replication. Having defined RPO & RTO. Off host backups. All without service interruption or degradation.
The good news is is that you might spend $1,000,000 on keeping all that going, but you won't be paying out tens of thousands of dollars a minute in salary and hundreds of thousands of dollars in lost business when the home brew solution explodes. And if, heaven forbid, the worst happens and your data center happens to burn to the ground, your DR site will be able to come online in ten minutes and every committed transaction will be sitting there on the drives ready for your apps, customers and employees.
Data centers already provide power and network - why don't they also provide shared storage?
The data center can buy several big mean SANs, and then provision storage sets to customers. Customers then only have to pay for what they use, can add space without installing new hardware, and get speed and reliability.
What SAN vendor is selling 15K 450GB SAS drives for $300? We pay roughly $2,000 per drive from EMC. We need to switch to whoever you guys are using!
"Modern" monied business is financially and "book" driven. Real service, real results are a bystance, a fortuitous coincidence. Oly the serfs care about them as if they were real. The rest is loans, leverages, collaterals - and bonusses, lots of bonusses. When something goes "poof!", it's on to the next one. It's a 3rd world inflationary spiral artificial economy thing.
What about controllers, trays, etc? What are these drives going into, servers with software raid? You can't just buy a bunch of 4U servers, stuff them full of drives and share them through iSCSI. That's how you run a porn site, not a data center. How do you handle drive failures, hot spares, etc?
What about the services from the vendor, licenses for enabling storage features like replication that usually is charged for TB of data replicated, licenses for the management software, etc, etc, etc... I would + 100.000 to your ballpark guess :-)
I used that backblaze analogy too, but after yours, @ano:
http://hardware.slashdot.org/comments.pl?sid=1735418&cid=33075574
It comes up to $45K/90TB tops, in just one 4U enclosure, management overhead and real estate included. ($50K/100TB)
To answer @spazimodo, the Taiwan-built Barracuda XT disks have low failure rates compared to the Seagate disks built in China, and you can combine them in RAID 6 or 60 with Adaptec controllers for $1-2K more. Why would you ever use RAID 5 anyway, that's insane. We're talking raw storage here, so backup hardware, snapshots, replication are exactly the same as the 1M/100TB raw space estimate. If you want real usable space data all costs included, get 2 such servers, let's say at $50K each with RAID 6 controllers and fiber optics cluster connection. That's 74TB of usable space, plus another 74TB on the backup clustered server, or $135K/100TB usable space.
Even a backblaze enclosure comprised exclusively of SSD disks for performance would cost only $72K per enclosure, hardware and 1st year costs included, about $640K/100TB. But I assume this level of performance is far above the $1M figure, assumedly for SAS disks, and which also probably spreads out the hardware cost onto 3 years, so it's more like $0.5M for pure SSD 100TB. It would really help if the article would detail that $1M cost.
SSD costs assuming 256GB disks @ $700 each. 1 backblaze 4U server would provide only 11TB of SSD raw space, 9TB of usable RAID 6 space.
It's quite bad that the $1M / 100TB cost is not detailed in any way in this article, it makes all attempts of comparison futile and impossible.
Thankfully several commenters provided their input, some even mentioning billing $5M per 100TB to their lawyers customers.
My guess is that this $1M represents partially the costs of the hardware/software configurations in existing datacenters, with most of these possibly purchased 2 years ago.
2 years is a pretty long time in IT, with many technological and financial changes, such as SATA III, 2TB disks and SSD, the latter still being rather immature and expensive.
Given the relatively low cost of the hardware, it does make sense to implement a better aggregation/allocation infrastructure, and disconnect the unused servers, but keep them on hold to add extra storage in a moment's notice, or to scavenge spare hard disks. They won't cost a dime if they are plugged but powered off, it's the online storage and maintenance that costs $1M/100TB (if that much.)
Fault Tolerance - at least in the market leading VMware world requires thick provisioning.
We have a few 100 TBs of SAN storage in our hospital. Some people would consider having the storage laid out as RAID 10 on each array and the production data base software mirrored across 3 arrays to be wasteful. Not really. Recently we had one of the data centers shutdown when maintenance was doing a generator test and forgot to put the generator in by-pass. The entire data center went dark, but the Epic application continued to run because of the mirroring done between data centers.
It took some time to clean up the mess that was caused, but the data base never went down.
As to the third mirror, we use that for nightly backups. The database is frozen in time, the mirror snapped off and moved to a different server. We backup the data base to tape, refresh a SUP{port} instance, and run integrity checks on the mirror. Once everything has completed, we bring the mirror (storage) back over to the production server and resync the volumes.
All in all, I'd say we have an HA solution that is pretty much bullet proof. It takes a lot of storage, but it gets the job done. As to waste, very little is actually wasted.
In the world of storage area networks you must design too support the IO load first and capacity will typically never be an issue - in tier one and two storage.
With cache cards and ssd becoming cheaper this rule is changing but for many SANs they have wasted space only because they needed more spindles to support the load.