Slashdot Mirror

← Back to Stories (view on slashdot.org)

Dropbox Moves Users' Data Off Amazon S3 to Its Own Infrastructure

Posted by msmash on from the boyhood-to-manhood dept.

Reader Richard_at_work writes: Dropbox today announced that it has been working on a "top secret" project called Magic Pocket for the past two and a half years to get data of more than 500 million users from Amazon S3 to its own custom-built infrastructure. The company says that it has migrated over 90% of its users' data so far. Dropbox's relationship with AWS isn't completely over, however, as they will continue to use AWS for specific regional data stores where there is a requirement.

3 of 45 comments (clear)

  1. Rolling your own is great until it fails by Overzeetop · · Score: 3, Insightful

    Because your own cloud server requires maintenance, and when your cloud server goes down you're SOL until you, personally, have the time to troubleshoot and fix it.

    How do I know this? My server developed a tic in it's network card, corrupting about 1 bit in every 5,000,000,000 or so. Took me a year to find that I actually had a problem with the server, and then two weeks to narrow down what the problem actually was. As a side effect I also found that I had a dodgy drive cable (one of 6 in the system) which showed no outward sign of problems because CRCs were correcting those bit problems.

    Could this happen to a cloud service? Sure. Are they likely to catch it? Faster than I am, in all likelihood. Will it take them less time to correct it? You're damn sure it will. And for the cost of the time I spent troubleshooting my server, I could have paid for a decade of service from two cloud services so that I had 100% redundancy, and still had money to go buy a kegerator so I could drink beer instead of chasing bit problems.

    --
    Is it just my observation, or are there way too many stupid people in the world?
    1. Re:Rolling your own is great until it fails by hawguy · · Score: 4, Insightful

      Because your own cloud server requires maintenance, and when your cloud server goes down you're SOL until you, personally, have the time to troubleshoot and fix it.

      How do I know this? My server developed a tic in it's network card, corrupting about 1 bit in every 5,000,000,000 or so. Took me a year to find that I actually had a problem with the server, and then two weeks to narrow down what the problem actually was. As a side effect I also found that I had a dodgy drive cable (one of 6 in the system) which showed no outward sign of problems because CRCs were correcting those bit problems.

      Could this happen to a cloud service? Sure. Are they likely to catch it? Faster than I am, in all likelihood. Will it take them less time to correct it? You're damn sure it will. And for the cost of the time I spent troubleshooting my server, I could have paid for a decade of service from two cloud services so that I had 100% redundancy, and still had money to go buy a kegerator so I could drink beer instead of chasing bit problems.

      Don't count on it being any easier to troubleshoot rare network glitches with a cloud provider. Admittedly most of the time you can just launch a new instance and the problem goes away, but not always.

      The first thing they'll do is close your ticket with "can not reproduce", so it'll be up to you to provide a test case to reproduce the problem. Which may not be trivial since you have limited visibility into their systems. And you have to convince them that it's not a security group problem, and not a local configuration problem (like iptables). And even then they may dismiss your ticket because you're not running their officially supported kernel version, so you'll have to fight with them to accept that it is a real problem, or capitulate and try to repro on their supported software version.

      It took me 6 months to convince AWS support that there was a rare bug in network setup (not all subnets were reachable) that only hit once ever 500 - 1000 instance launches. They finally admitted that it was some sort of rare convergence problem in their network stack and that they are not monitoring for such problems so it won't recur.

      At least when you own the hardware, you have full visibility into the entire stack, and while you can sitll have different teams pointing the finger at each other, they all work for the same company so management can step in and tell them to stop pointing fingers and work together to find the solution.

  2. Re:Can I download my files as a .zip archive yet? by Richard_at_work · · Score: 4, Insightful

    Isn't that the whole goddamn point of the cloud? I can just use my goddamn web browser to interact with it, instead of a custom native app?!

    Uh, no. The cloud is whatever the people running the cloud want it to be, you just want it to be something different - there are no rules regarding what the cloud must do.

    At the end of the day, Dropbox is a syncing platform - that "goddamn desktop client" is the entire thing Dropbox is built around. If you wanted a different feature set, you chose the wrong product to use - there's no shame in admitting that, just don't blame the tool.

    Dropbox has issues creating zip files for huge data sets, because it doesn't want to commit a massive amount of resources to building that zip file, its as simple as that - if that's the way you are using Dropbox, then you are using it wrongly and not as its intended to be used.