Slashdot Mirror
How To Add 5.5 Petabytes and Get Banned From Costco
concealment writes with this extract from GigaOm: "'We buy lots and lots of hard drives . . . . [They] are the single biggest cost in the entire company.' Those are the words of Backblaze Founder and CEO Gleb Budman, whose company offers unlimited cloud backup for just $5 a month, and fills 50TB worth of new storage a day in its custom-built, open source pod architecture. So one might imagine the cloud storage startup was pretty upset when flooding in Thailand caused a global shortage on internal hard drives last year. Backblaze details much the process in a Tuesday-morning blog post, including the hijinks that followed as the company got creative trying to figure out ways around the new hard drive limits. Maps were drawn, employees were cut off from purchasing hard drives at Costco — both in-person throughout Silicon Valley and online (despite some great efforts to avoid detection, such as paying for hard drives online using gift cards) — and friends and family across the country were conscripted into a hard-drive-buying army."
Unlimited storage for $5/mo? I have to get on this shit.
.. buy direct or maybe some wholesale? Is such deliberate effort worth the actual cost?
Seriously, what a bunch of assholes.
So instead of doing the capitalistic thing and gouging with insanely high prices, the shops instead started rationing drives for a sane price so everyone could get a little bit of the very limited supply.
That was actually a really good thing to do. Instead of profiteering, they tried to make the best of a bad situation for everyone.
Then a bunch of dicks like this figure that they're more important than everyone else and that they should be able to get more than enyone else.
Selfish bastards. Nothing but scum.
After reading this I will not be giving them my money.
SJW n. One who posts facts.
Hear the story direct from Backblaze (bonus: goes into more detail).
The real litigious bastards...
I'm confused. Was Costco selling these drives at a loss or something, just to get people in the door?
I can't think of many good reasons that they would look at customers coming in and buying assloads of their merchandise and say "NO! Get out of here and don't buy stuff from us ever again!"
Porquoi?
There's a link in TFA to an older article about them starting up and how much it costs them to run their storage: apparently it costs them $94,563 for hardware, space and power for a petabyte over 3 years. That's about $0.0026 per gigabyte per month, so if users are charged $5 per month then they just have to gamble that the average user is going to use less than about 1.9 terabytes per month.
Obviously that doesn't include a lot of other stuff like wages, but you're still gambling that a user isn't gonna have more than a terabyte of data in the system, which doesn't seem like an unreasonable gamble for now.
... is pretty cheap (5$ is for a family account). But as BB itself says, you can only upload 2 to 4 GB per day.
They should be making a mint on that service! They use home-brew storage pods and are very open about it, too!
http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/
Anyway, be careful to read all the gotchas:
http://www.backblaze.com/remote-backup-everything.html (hint: 'everything' for a certain definition of everything. No virtual machines, ISO's and NAS storage by default.)
http://www.backblaze.com/internet-backup.html (hint: not all OSes are treated equally.)
(Full disclosure: I work for a storage manufacturer that sells de-duping storage so I think I understand their cost model a bit better than most.)
Karma? What's that again?
A backup in your basement does nothing for you if your house burns down/gets flooded/has a catastrophic power surge/whatever.
Where else can you backup offsite?
--PM
Several months ago I met someone from the Internet Archive (archive.org) who told a similar story. The weren't expanding their storage at the same pace as Backblaze, but they were also resorting to shucking external drives to build their rack mounted servers.
Religion is poison to rationality, and we lose sight of that at our own peril. -- Lurker2288
Yeah. I'll bet they're not even using oxygen-free SATA cables either.
Who cares what they store it on? What's important is it adequately checked for consistency, and what are the backups like. Everything else is detail.
Guess what. Google bought off-the-shelf computer gear for years and some datacentres run things without "datacentre grade" cooling. They don't suffer because a) they do it properly (i.e. not RELY on those drives to never fail) and b) nobody notices because the service is still more than good enough.
"Enterprise"-grade drives are just warrantied for longer. It doesn't mean they won't die just as quickly. Like "RAID"-grade drives - same drive as every other one on the production lines.
It's like saying you can't use Intel Mobile chips in a datacenter. It might not be your first choice, but provided they fulfil all their service obligations (which includes response times, failover, etc.) then who notices and who cares?
Every single server I've ever installed used "consumer grade" drives. Every single desktop I've ever installed used "consumer grade" drives. Failures are actually FAR BELOW any stated MTBF and, who cares, because it takes seconds to replace and DOES NOT AFFECT THE OVERALL SERVICE for the user. And no-one I've worked for has ever lost data because of a drive failure. Ever. Even when servers have all but caught fire.
It actually makes good sense as part of a complete backup system.
What happens to your data when your office/house/whatever with the 2 or 3 TB drives burns down with them in it, or someone breaks in and steals your desktop and the USB drive you left sitting on top of it?
Depending on the circumstances, I usually recommend RAID of some kind if possible, a USB/External Hard Drive on-site, and then some kind of off-site backup.
If your internal drive dies, if you had RAID, you just replace the dead drive. If no RAID, then you restore from your External backup. If you had a fire/theft or other major loss, you restore from web/off-site. In the case of BackBlaze, they offer 3 restoration options, included zip download of files, or FedEx thumb drive or external drive for additional cost.
You should read on how they build their systems. One of the ways they keep costs so low is using consumer grade hardware with the idea that it will fail. In general, consumer grade hard drives have about the same failure rate as "enterprise grade", they just usually have lower transfer rates. When your clients are syncing over 768k DSL uploads or even 3-5 Mbps cable upload speeds, hard drive speed is not going to be your bottleneck.
They actually have a guy whose job it is to just go around a day or two a week through their data center and replace the dead drives. Due to the redundancy they built into their systems, a drive failure isn't a big deal or really unexpected.
photography eh?
Nudge, Nudge, wink wink
Say no more
I think I had a time limited error recovery bit trying to read that post.
Tic-Tac-Toe, Global Thermonuclear War, and relationships all have the same winning move.
Costco has a corporate policy that limits revenue from sales to 10% above their cost. This 10% covers their overhead costs (buildings,employees, distribution, etc). 100% of Costco's profit comes from their membership fees. Depending on the amount of fuel sold per quarter they may turn a very small profit on this 10% or they might not.
Costco has NO profit incentive to sell one customer more of a product if that means pissing off other customers. Their profits come almost exclusively from membership fees, hence the drive to get everyone signed up for executive memberships.
Costco generally limits markup to 15%, not 10%. Also, certain state laws require that Costco apply minimum markups to the selling prices for specific goods, such as tobacco products, alcoholic beverages, and gasoline. Of course, some products are marked down for quick sale. However, the resultant average gross margin target is around 10%.
They do, however, attempt to control their SG&A (overhead) to match their gross margin target of 10%. The net corporate profit is from membership fees which is why they try so hard to get you to sign up for executive memberships...
I'm not sure who you are disagreeing with, but it's not me. My point was that there is a difference between consumer and enterprise grade drives. Not that enterprise grade is more reliable than consumer grade. The main differences I listed have to do with performance and the ability to get one's data off the drive.
Here are some basic facts about storage for you. A 5400 RPM drive has an average rotational latency of 5.5 milliseconds. A 7200 RPM drive averages 4 milliseconds. A 10K RPM drive, 3 milliseconds. And 15K RPM averages 2 milliseconds. This is just basic math -- any particular block on the drive will sometimes be right before the head and sometimes right behind the head. On average the platter has to spin halfway around to bring that block to the head.
But that's only a portion of the performance of a drive. The other part is how long it takes the head to move from track to track. This is much more design dependent. But in general enterprise drives are expected to have a seek time in the 3-5 millisecond range and consumer drives run 5 and up, typically 5-10 milliseconds.
Add these up and a typical 15K RPM drive will have about 6 ms latency and a typical 7200 RPM drive will have 11 ms latency. Which means that a 15K RPM drive can do approximately 165 random IO operations at it's typical latency (normally measured in terms of 4k, 8k or 16k IOs.) A 7200 RPM drive can do approximately 90 random IOPS. This is a big deal when dealing with multi-user server applications.
Additionally, all SAS and FC drives are dual ported and SAS and FC fabrics are multi-initiator. Which allows them to be deployed in fully redundant and fully active configurations (two paths between server and array, two controllers in the array, two mirrored caches and two paths from each controller to each disk.) A SATA drive has one port. There are port multiplexers that can be inserted between the drive and the chassis, but because the drive itself is natively single ported, only one of the multiplexed ports can be active at a time and thus are limited to having fail-over between the controllers rather than active-active controllers.
As far as RAID performance goes... Two mirrored 7200 RPM drives do not provide the equivalent of a 14.4K RPM drive. Minimum latency is limited by the speed of a single drive regardless of RAID of any type. Here's what a two drive RAID1 gets you: one redundant copy of your data. Twice as many read operations at the same latency as a single disk. And the write performance of a single disk. Because you can do twice as many read operations, you get double the read bandwidth. Yes you can add more drives to your mirror, but there comes a point where the rest of your storage subsystem becomes less redundant that your drives. RAID5 (or RAID6, or RAIDZ, or RAIDZ2, etc.) gets you redundancy to the level of however many disks worth of parity your system implements. For a standard RAID5 that is a single disk failure, for RAID6 it's two disks, etc. Read performance increases as a multiple of the number of drives in your raid group. Write performance is a read and a write of your data block plus a read and write for each parity block. For RAID5 that means that each write will do four IO operations into the raid group. So an eight drive raid group should get double the write performance of a single drive. Of course any array that one would use in an enterprise environment will have at least two battery backed up caches, which makes any write penalties moot as all writes are cached.
As far as reliability goes, that's an interesting question. The fact is hard drives die. However the premium I pay buying hard drives from my storage or server vendor includes 4 hour replacement SLAs in western countries and in less developed areas it's typically 24 hours. I don't know what Costco's policy is, but I'm sure it doesn't involve bringing the HDD to me today and I'd be surprised if I could show up three years later and have them replace my HDD with a matching device. Additionally consumer grade drive