How To Add 5.5 Petabytes and Get Banned From Costco

← Back to Stories (view on slashdot.org)

How To Add 5.5 Petabytes and Get Banned From Costco

Posted by timothy on Tuesday October 9, 2012 @02:50AM from the you-need-the-executive-membership dept.

concealment writes with this extract from GigaOm: "'We buy lots and lots of hard drives . . . . [They] are the single biggest cost in the entire company.' Those are the words of Backblaze Founder and CEO Gleb Budman, whose company offers unlimited cloud backup for just $5 a month, and fills 50TB worth of new storage a day in its custom-built, open source pod architecture. So one might imagine the cloud storage startup was pretty upset when flooding in Thailand caused a global shortage on internal hard drives last year. Backblaze details much the process in a Tuesday-morning blog post, including the hijinks that followed as the company got creative trying to figure out ways around the new hard drive limits. Maps were drawn, employees were cut off from purchasing hard drives at Costco — both in-person throughout Silicon Valley and online (despite some great efforts to avoid detection, such as paying for hard drives online using gift cards) — and friends and family across the country were conscripted into a hard-drive-buying army."

35 of 273 comments (clear)

Min score:

Reason:

Sort:

Wow by taktoa · 2012-10-09 02:55 · Score: 4, Insightful

Unlimited storage for $5/mo? I have to get on this shit.
1. Re:Wow by vlm · 2012-10-09 03:10 · Score: 5, Interesting
  
  Unlimited storage for $5/mo? I have to get on this shit.
  Run the numbers. 50 TB a day sounds like a lot, if they've only got one customer. But they're probably got "a zillion" which would imply your very thin slice of the upload bandwidth is going to be choked to like a gig per day. The upgrade in my basement from (full) 1 TB drives to 2 TB drives took around overnight, less than 24 hours anyway, but over the net at a gig per day would be about 3 years to fully convert. Even if they're not limited I would have serious problems shoving more than 100 gigs/day thru my cablemodem, so thats at least 10 days.
  Another interesting thing to analyze is $5/month is $60/yr, so subtract $5/yr for electricity to spin a drive, assume a drive lives 2 years (probably much longer) that means if you can buy a drive big enough to hold everything you want for less than $110, just stick a drive in your basement. Better bandwidth and latency too, I have gigE at home but only ten or so megs of cablemodem. $110 at tiger direct will get me 2 TB. So 2 TB is approximately the tipping point, use less and you're better off "self hosting" in the basement, use more and you're better off using their service (and they're likely losing money if you use more than 2 TB).
  Also I'm curious if its "unlimited" like cellphone or internet access is "unlimited" in other words they'll cut you off if they're losing money on you.
  
  --
  "Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
2. Re:Wow by wonkey_monkey · 2012-10-09 03:20 · Score: 5, Insightful
  
  Who says Slashvertising doesn't work?
  
  --
  systemd is Roko's Basilisk.
3. Re:Wow by fuzzyfuzzyfungus · 2012-10-09 03:25 · Score: 5, Insightful
  
  Unlimited storage for $5/mo? I have to get on this shit.
  Website says $3.96/m for unlimited data.
  Something tells me this business model will not survive without some serious bandwidth limitations. After all, if you upload is limited to 100mb then you ability to (non commercially) fill Terabytes of data is limited.
  My impression(from friends who use them) is that they aim pretty heavily at home-user backup scenarios who are likely to be comparatively light users and have severely limited upstream bandwidth. They also don't do Big Serious SLAs and similar. Nor do they support things like backing up mounted NAS volumes or non Windows/OSX systems(I haven't check to see if the client is smart enough to recognize a mounted iSCSI device... It isn't exactly rocket surgery to distinguish a block device hanging from the Windows iSCSI initiatior from a block device hanging off the Intel whateverchipset SATA 2 port; but if you go with 'NAS = SMB/AFP" you'd miss it.
  Still, convenient and cheap, if not as robust as solutions that cost more.
4. Re:Wow by fuzzyfuzzyfungus · 2012-10-09 06:15 · Score: 3, Funny
  
  Indeed. here's an article about some rocket surgeons.
5. Re:Wow by im_thatoneguy · 2012-10-09 06:33 · Score: 3, Interesting
  
  I have a RED camera which shoots 18megapixel raw photos at 24+ frames per second. Backblaze hasn't throttled me at all and I have 20mbps upload speeds.
6. Re:Wow by Bryansix · 2012-10-09 07:39 · Score: 3, Informative
  
  Yes, I'm a photographer and I have 5TB of storage in my computer and I use Backblaze. I recently had a 1TB hard drive die (controller failure). So I began to restore to my ample space on my other drives. The problem I ran into is that the restore file only gets hosted for 7 days and then deleted. So in order to restore I have to break up the restore into smaller bites and download them one at a time. My other option is that they will ship me an external 1TB drive for $189. I'm not on my fourth file download of the restore. I'll get all my files back but its a serious pain.
7. Re:Wow by JTsyo · 2012-10-09 08:25 · Score: 5, Funny
  
  I have a BLACK camera, what does that have to do with anything?
Can't they just... by grumpyman · 2012-10-09 02:56 · Score: 3, Insightful

.. buy direct or maybe some wholesale? Is such deliberate effort worth the actual cost?
1. Re:Can't they just... by 140Mandak262Jamuna · 2012-10-09 03:03 · Score: 4, Informative
  
  There was flooding in Thailand. Factories were disturbed. This company tried to grab as much of the drives already in the pipeline as it could.
  
  --
  sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
2. Re:Can't they just... by drinkypoo · 2012-10-09 03:05 · Score: 3, Interesting
  
  Costco gets such a good deal by buying in massive lots and handling distribution themselves that in fact it is worth the actual cost. I've been a costco customer off and on over the last few years and by remarkable non-coincidence I pretty much always check HDD prices. As well, I bought a pair of 1TB MyBooks when they were the big disk that Costco sold, and more recently I've bought a pair of 3TB GoFlexes. I like the return policy, and I can back up one disk to another (I disco the backup) and the two disks have very different service profiles so they're not entirely likely to die at the same time. And the prices were significantly lower than anything I could find from an even vaguely reputable retailer online.
  People have complained bitterly about how hard it supposedly is to get a disk out of a MyBook, I've never tried. I don't know anything at all about how hard it is to get the disk out of a GoFlex enclosure. But I do know that until very recently the prices on these external disks were actually better than buying internal disks online. I know (personally) a couple of people who went that route when building a desktop system, decasing externals from costco.
  
  --
  "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
3. Re:Can't they just... by the_B0fh · 2012-10-09 03:46 · Score: 5, Informative
  
  some hard drives (western digital, iirc) are now sold without the sata interface on the drive itself, for external models.
  you rip it apart, and find out that you can't stick it onto a sata port...
What a bunch of douche bags by serviscope_minor · 2012-10-09 03:03 · Score: 5, Interesting

Seriously, what a bunch of assholes.
So instead of doing the capitalistic thing and gouging with insanely high prices, the shops instead started rationing drives for a sane price so everyone could get a little bit of the very limited supply.
That was actually a really good thing to do. Instead of profiteering, they tried to make the best of a bad situation for everyone.
Then a bunch of dicks like this figure that they're more important than everyone else and that they should be able to get more than enyone else.
Selfish bastards. Nothing but scum.
After reading this I will not be giving them my money.

--
SJW n. One who posts facts.
1. Re:What a bunch of douche bags by serviscope_minor · 2012-10-09 03:29 · Score: 4, Insightful
  
  BackBlaze did what they should have done: solve the business problem at hand.
  Ah, choice use of the word "business". That must mean that it's OK to suspend all morals as long as it's "business".
  Basically no.
  BlackBlaze did nothing. Do not pretentd otherwise by using the word "business" to hide the acts of individuals.
  The _people_ at balckblaze figured that they they would be selfish and put their needs above the needs of everyone else. That's selfish, douchebaggy behaviour.
  One that Costco, by not gouging with insane prices, wasn't engaing in.
  
  --
  SJW n. One who posts facts.
2. Re:What a bunch of douche bags by Rob+the+Bold · 2012-10-09 03:51 · Score: 3, Interesting
  
  Seriously, what a bunch of assholes.
  So instead of doing the capitalistic thing and gouging with insanely high prices, the shops instead started rationing drives for a sane price so everyone could get a little bit of the very limited supply.
  That was actually a really good thing to do. Instead of profiteering, they tried to make the best of a bad situation for everyone.
  Then a bunch of dicks like this figure that they're more important than everyone else and that they should be able to get more than enyone else.
  Selfish bastards. Nothing but scum.
  After reading this I will not be giving them my money.
  The hard drive sellers weren't doing this altruistically. They made their call, figuring that rationing drives was the best for their businesses and best fulfilled their duty to their shareholders.
  The cloud storage company also did what they thought best benefited their business and fulfilled their duty to shareholders/backers/etc.
  It does not benefit us as individuals to assign a moral motive to the actions of a company. Whatever they do, it's for a business purpose. If it seems like one company is the good guy, it's just that that's what they think will help them return value to the owners. We must realize that they are all "selfish bastards" by the very nature of the capitalist system and not be fooled into personifying them.
  What I've written in no way implies that you can't spend your money wherever you want -- or withhold it -- based on anything you want to base it on. If you find a business' actions to be detrimental to society or just contrary to your ideals, you can certainly boycott them. In fact, I think you should boycott if you feel as strongly as you appear to. In this way -- if you're not alone -- those actions on the part of the company may turn out not to benefit the shareholders and therefore force them to change.
  You could also pursue a legislative approach, maybe convince your representatives that legal rationing is in the public interest. The US rationed commodities during wartime before, perhaps you can appeal that adequate storage space for all is sufficiently important for legal intervention.
  
  --
  I am not a crackpot.
3. Re:What a bunch of douche bags by AlgUSF · 2012-10-09 03:52 · Score: 4, Informative
  
  Honestly, I wouldn't think twice about doing the same thing. They are purchasing the drives, not stealing them. For "some reason" costco is buying them in lots where they can distribute them at that price. I guess they were just leveraging Costco and Best Buy's buying power to keep their business afloat.
  
  --
  
  I want my rights back. I was actually using them when our government stole them after 9/11.
4. Re:What a bunch of douche bags by iluvcapra · 2012-10-09 04:04 · Score: 3, Interesting
  
  I agree that it's a pretty selfish thing to do, although I would probably have done the same in their place.
  The magic of Adverse Selection, when you discover the Golden Rule has a discount rate...
  
  --
  Don't blame me, I voted for Baltar.
5. Re:What a bunch of douche bags by serviscope_minor · 2012-10-09 04:21 · Score: 3, Insightful
  
  The hard drive sellers weren't doing this altruistically. They made their call, figuring that rationing drives was the best for their businesses and best fulfilled their duty to their shareholders.
  they almost certainly could have got away with upping the prices by much more than they did. They chose strict rationing instead.
  It does not benefit us as individuals to assign a moral motive to the actions of a company. Whatever they do, it's for a business purpose.
  And this is a real problem. They're just Soylent Corporations: made of people. It's the people that act. Pretending otherwise is an attempt to justify imorral action.
  Taking that to its illoigcal conclusion could lead to hypotheticals like
  Well, it's OK that they kidnapped babies, murdered them and sold them for dog food. It was a really cheap source of meat and so the profit margin was immense. Excellent for business.
  We must realize that they are all "selfish bastards" by the very nature of the capitalist system and not be fooled into personifying them.
  No: you have been fooled into believeing that companies act independent of people. They do not and they cannot. Only people can act and people can and do choose to do immoral things for money.
  The fact that they're doing it for money is no justification whatsoever.
  
  --
  SJW n. One who posts facts.
6. Re:What a bunch of douche bags by neminem · 2012-10-09 05:01 · Score: 3, Interesting
  
  Well, that -is- how it works, sadly... Great quote from this season of Leverage:
  "I mean, you and I kill a guy, we go to prison... my company kills a guy, we pay a fine."
Skip the blogspam by maztuhblastah · 2012-10-09 03:03 · Score: 4, Informative

Hear the story direct from Backblaze (bonus: goes into more detail).

--
The real litigious bastards...
I don't get it by JobyOne · 2012-10-09 03:06 · Score: 4, Interesting

I'm confused. Was Costco selling these drives at a loss or something, just to get people in the door?
I can't think of many good reasons that they would look at customers coming in and buying assloads of their merchandise and say "NO! Get out of here and don't buy stuff from us ever again!"

--
Porquoi?
1. Re:I don't get it by xlsior · 2012-10-09 03:14 · Score: 5, Insightful
  
  Many companies reserve the right to limit quantities. Making one customer happy by selling them every drive in stock means ticking off hundreds of others that wouldn't be able to buy the single drive they need.
2. Re:I don't get it by Anonymous Coward · 2012-10-09 03:15 · Score: 4, Insightful
  
  I'm confused. Was Costco selling these drives at a loss or something, just to get people in the door?
  I can't think of many good reasons that they would look at customers coming in and buying assloads of their merchandise and say "NO! Get out of here and don't buy stuff from us ever again!"
  A valid question, but one which some logical thought should provide an answer to... I'd suspect Costco prefers to have many content customers (a customer who ends up at an empty shelf every day is going to go elsewhere, potentially even for other stuff) than one deliriously happy customer. The profit margins on those things are going to be minor anyway, so its not Costco were raking in the profits by selling all their harddrive stock. Presumably, this added profit did not offset all the other customers being unhappy with Costco that they couldnt buy they harddrives they advertise.
3. Re:I don't get it by localman57 · 2012-10-09 03:19 · Score: 5, Insightful
  
  I'm confused. Was Costco selling these drives at a loss or something, just to get people in the door?
  There's a difference between selling at a loss, and selling below market value. For instance, if Costco signs a contract for delivery of a million drives in Feburary, the factory floods in March, and Costco gets delivery in April, their drives are suddenly worth substantially more. They can either sell them at the previously intended prices, or they can raise prices to market value. In the first case they still sell them for more than they paid, but less than market value. In the second case, they take the customer for all they're worth, and make much more profit. Rationing is the only way the first one can work, otherwise someone will come in and buy all your drives, then resell them at market value.
4. Re:I don't get it by BetterSense · 2012-10-09 04:06 · Score: 3, Funny
  
  Unless the drives are copyrighted!
3.96$ a month... by geschild · 2012-10-09 03:14 · Score: 3, Interesting

... is pretty cheap (5$ is for a family account). But as BB itself says, you can only upload 2 to 4 GB per day.
They should be making a mint on that service! They use home-brew storage pods and are very open about it, too!
http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/
Anyway, be careful to read all the gotchas:
http://www.backblaze.com/remote-backup-everything.html (hint: 'everything' for a certain definition of everything. No virtual machines, ISO's and NAS storage by default.)
http://www.backblaze.com/internet-backup.html (hint: not all OSes are treated equally.)
(Full disclosure: I work for a storage manufacturer that sells de-duping storage so I think I understand their cost model a bit better than most.)

--
Karma? What's that again?
House burns down? by PeterM+from+Berkeley · 2012-10-09 03:28 · Score: 4, Insightful

A backup in your basement does nothing for you if your house burns down/gets flooded/has a catastrophic power surge/whatever.
Where else can you backup offsite?
--PM
1. Re:House burns down? by philipmather · 2012-10-09 03:40 · Score: 5, Funny
  
  I would say at your parent's house but this being slashdot that's probably not offsite.
  
  --
  Regards, Phil
2. Re:House burns down? by xaxa · 2012-10-09 03:53 · Score: 4, Interesting
  
  I would say at your parent's house
  That's exactly what I've done. I set up some scripts to rsync data from my computer to a server in my mum's garage, and also the reverse.
  That way, we both have important data (mostly photos) backed up off-site in different cities, and the photos are available to browse through a web interface.
  
  but this being slashdot that's probably not offsite.
  A friend went with an encrypted backup program, and set up more-or-less the same thing with another friend.
3. Re:House burns down? by JazzLad · 2012-10-09 05:41 · Score: 4, Insightful
  
  3 words:
  
  Safe Deposit Box.
  
  I have 3 drives (that are archive only, not OS), 2 in my PC and 1 in my bank. Every couple weeks or so I take one out of my PC & swap it at the bank & update the bank's drive to mirror the one still in my PC. Cost is ~30/yr for the box except that I already had it for documents anyway (so no adtl cost to me).
  
  --
  "If you have nothing to hide, you have nothing to fear." - Every fascist, ever
Internet Archive by dr_leviathan · 2012-10-09 03:28 · Score: 4, Informative

Several months ago I met someone from the Internet Archive (archive.org) who told a similar story. The weren't expanding their storage at the same pace as Backblaze, but they were also resorting to shucking external drives to build their rack mounted servers.

--
Religion is poison to rationality, and we lose sight of that at our own peril. -- Lurker2288
Re:All on consumer grade drives..... by ledow · 2012-10-09 03:49 · Score: 3, Informative

Yeah. I'll bet they're not even using oxygen-free SATA cables either.
Who cares what they store it on? What's important is it adequately checked for consistency, and what are the backups like. Everything else is detail.
Guess what. Google bought off-the-shelf computer gear for years and some datacentres run things without "datacentre grade" cooling. They don't suffer because a) they do it properly (i.e. not RELY on those drives to never fail) and b) nobody notices because the service is still more than good enough.
"Enterprise"-grade drives are just warrantied for longer. It doesn't mean they won't die just as quickly. Like "RAID"-grade drives - same drive as every other one on the production lines.
It's like saying you can't use Intel Mobile chips in a datacenter. It might not be your first choice, but provided they fulfil all their service obligations (which includes response times, failover, etc.) then who notices and who cares?
Every single server I've ever installed used "consumer grade" drives. Every single desktop I've ever installed used "consumer grade" drives. Failures are actually FAR BELOW any stated MTBF and, who cares, because it takes seconds to replace and DOES NOT AFFECT THE OVERALL SERVICE for the user. And no-one I've worked for has ever lost data because of a drive failure. Ever. Even when servers have all but caught fire.
Ob python by rossdee · 2012-10-09 04:07 · Score: 3, Funny

photography eh?
Nudge, Nudge, wink wink
Say no more
Re:All on consumer grade drives..... by oodaloop · 2012-10-09 04:14 · Score: 3, Funny

I think I had a time limited error recovery bit trying to read that post.

--
Tic-Tac-Toe, Global Thermonuclear War, and relationships all have the same winning move.
Re:All on consumer grade drives..... by Score+Whore · 2012-10-09 08:42 · Score: 3, Interesting

I'm not sure who you are disagreeing with, but it's not me. My point was that there is a difference between consumer and enterprise grade drives. Not that enterprise grade is more reliable than consumer grade. The main differences I listed have to do with performance and the ability to get one's data off the drive.
Here are some basic facts about storage for you. A 5400 RPM drive has an average rotational latency of 5.5 milliseconds. A 7200 RPM drive averages 4 milliseconds. A 10K RPM drive, 3 milliseconds. And 15K RPM averages 2 milliseconds. This is just basic math -- any particular block on the drive will sometimes be right before the head and sometimes right behind the head. On average the platter has to spin halfway around to bring that block to the head.
But that's only a portion of the performance of a drive. The other part is how long it takes the head to move from track to track. This is much more design dependent. But in general enterprise drives are expected to have a seek time in the 3-5 millisecond range and consumer drives run 5 and up, typically 5-10 milliseconds.
Add these up and a typical 15K RPM drive will have about 6 ms latency and a typical 7200 RPM drive will have 11 ms latency. Which means that a 15K RPM drive can do approximately 165 random IO operations at it's typical latency (normally measured in terms of 4k, 8k or 16k IOs.) A 7200 RPM drive can do approximately 90 random IOPS. This is a big deal when dealing with multi-user server applications.
Additionally, all SAS and FC drives are dual ported and SAS and FC fabrics are multi-initiator. Which allows them to be deployed in fully redundant and fully active configurations (two paths between server and array, two controllers in the array, two mirrored caches and two paths from each controller to each disk.) A SATA drive has one port. There are port multiplexers that can be inserted between the drive and the chassis, but because the drive itself is natively single ported, only one of the multiplexed ports can be active at a time and thus are limited to having fail-over between the controllers rather than active-active controllers.
As far as RAID performance goes... Two mirrored 7200 RPM drives do not provide the equivalent of a 14.4K RPM drive. Minimum latency is limited by the speed of a single drive regardless of RAID of any type. Here's what a two drive RAID1 gets you: one redundant copy of your data. Twice as many read operations at the same latency as a single disk. And the write performance of a single disk. Because you can do twice as many read operations, you get double the read bandwidth. Yes you can add more drives to your mirror, but there comes a point where the rest of your storage subsystem becomes less redundant that your drives. RAID5 (or RAID6, or RAIDZ, or RAIDZ2, etc.) gets you redundancy to the level of however many disks worth of parity your system implements. For a standard RAID5 that is a single disk failure, for RAID6 it's two disks, etc. Read performance increases as a multiple of the number of drives in your raid group. Write performance is a read and a write of your data block plus a read and write for each parity block. For RAID5 that means that each write will do four IO operations into the raid group. So an eight drive raid group should get double the write performance of a single drive. Of course any array that one would use in an enterprise environment will have at least two battery backed up caches, which makes any write penalties moot as all writes are cached.
As far as reliability goes, that's an interesting question. The fact is hard drives die. However the premium I pay buying hard drives from my storage or server vendor includes 4 hour replacement SLAs in western countries and in less developed areas it's typically 24 hours. I don't know what Costco's policy is, but I'm sure it doesn't involve bringing the HDD to me today and I'd be surprised if I could show up three years later and have them replace my HDD with a matching device. Additionally consumer grade drive