Dropbox Moves Users' Data Off Amazon S3 to Its Own Infrastructure

Reader Richard_at_work writes: Dropbox today announced that it has been working on a "top secret" project called Magic Pocket for the past two and a half years to get data of more than 500 million users from Amazon S3 to its own custom-built infrastructure. The company says that it has migrated over 90% of its users' data so far. Dropbox's relationship with AWS isn't completely over, however, as they will continue to use AWS for specific regional data stores where there is a requirement.

Comments

Not surprised

[vux984]

I'd say there is no surprise to see them vertically integrate; they're large enough to leverage the economies of scale of running their own storage for themselves; rather than to pay someone else to do it.

Can I download my files as a .zip archive yet?

[Anonymous Coward]

Does this mean that I'll finally be able to download my files as a .zip archive? I have some directories in Dropbox with a lot of files in them, and I get some bullshit message about the folder being too large to download, or something like that, when I try to use the functionality that exports the directory a .zip archive. It's not even that much data. Maybe like 5 GB in total. But I always get that fucking message, and it never lets me download these directories as an archive. I even bought the pro subscription, and it still won't let me easily download an archive of my directories! I don't want to install the goddamn desktop client just to copy a few directories of files from Dropbox! Isn't that the whole goddamn point of the cloud? I can just use my goddamn web browser to interact with it, instead of a custom native app?! Holy fuck, all I want to do is download an archive of a directory in Dropbox. Why the fuck do they make it impossible to do that easily?! Does this move to their own infrastructure finally make it possible for them to let me download my directories as .zip archives?!

Re:Can I download my files as a .zip archive yet?

[omnichad]

That's likely a limitation of the .zip format and/or how they generate them for export. It's hard to generate .zip data on-the-fly because the file has headers at the end as well as the beginning, so they are probably generating an actual file and storing it before sending. They don't want to store .zip files for large folders for long enough for you to download. Not to mention the CRC32 checksums required for the format are still computationally intensive at a large scale.

I'm sure it's possible to generate .zip data on-the-fly, but it would be a much more complex system than what they're using.

The whole point of their software is the desktop client. If you don't want to use that even to download a folder, it's probably the wrong tool for you.

Re:Can I download my files as a .zip archive yet?

[Richard_at_work]

Isn't that the whole goddamn point of the cloud? I can just use my goddamn web browser to interact with it, instead of a custom native app?!

Uh, no. The cloud is whatever the people running the cloud want it to be, you just want it to be something different - there are no rules regarding what the cloud must do.

At the end of the day, Dropbox is a syncing platform - that "goddamn desktop client" is the entire thing Dropbox is built around. If you wanted a different feature set, you chose the wrong product to use - there's no shame in admitting that, just don't blame the tool.

Dropbox has issues creating zip files for huge data sets, because it doesn't want to commit a massive amount of resources to building that zip file, its as simple as that - if that's the way you are using Dropbox, then you are using it wrongly and not as its intended to be used.

Why use some one elses cloud when you can make

[mmiscool]

Why use some one else cloud when you could make your own?

So, reinventing the wheel again

[OverlordQ]

So they basically re-did everything that backblaze did for it's storage pods.

ZIP has 4GB, 65535 file limit, needs extensions

[Anonymous Coward]

Unless they get new software that supports the ZIP64 extension probably not.

Rolling your own is great until it fails

[Overzeetop]

Because your own cloud server requires maintenance, and when your cloud server goes down you're SOL until you, personally, have the time to troubleshoot and fix it.

How do I know this? My server developed a tic in it's network card, corrupting about 1 bit in every 5,000,000,000 or so. Took me a year to find that I actually had a problem with the server, and then two weeks to narrow down what the problem actually was. As a side effect I also found that I had a dodgy drive cable (one of 6 in the system) which showed no outward sign of problems because CRCs were correcting those bit problems.

Could this happen to a cloud service? Sure. Are they likely to catch it? Faster than I am, in all likelihood. Will it take them less time to correct it? You're damn sure it will. And for the cost of the time I spent troubleshooting my server, I could have paid for a decade of service from two cloud services so that I had 100% redundancy, and still had money to go buy a kegerator so I could drink beer instead of chasing bit problems.

Re:Rolling your own is great until it fails

[hawguy]

Because your own cloud server requires maintenance, and when your cloud server goes down you're SOL until you, personally, have the time to troubleshoot and fix it.

How do I know this? My server developed a tic in it's network card, corrupting about 1 bit in every 5,000,000,000 or so. Took me a year to find that I actually had a problem with the server, and then two weeks to narrow down what the problem actually was. As a side effect I also found that I had a dodgy drive cable (one of 6 in the system) which showed no outward sign of problems because CRCs were correcting those bit problems.

Could this happen to a cloud service? Sure. Are they likely to catch it? Faster than I am, in all likelihood. Will it take them less time to correct it? You're damn sure it will. And for the cost of the time I spent troubleshooting my server, I could have paid for a decade of service from two cloud services so that I had 100% redundancy, and still had money to go buy a kegerator so I could drink beer instead of chasing bit problems.

Don't count on it being any easier to troubleshoot rare network glitches with a cloud provider. Admittedly most of the time you can just launch a new instance and the problem goes away, but not always.

The first thing they'll do is close your ticket with "can not reproduce", so it'll be up to you to provide a test case to reproduce the problem. Which may not be trivial since you have limited visibility into their systems. And you have to convince them that it's not a security group problem, and not a local configuration problem (like iptables). And even then they may dismiss your ticket because you're not running their officially supported kernel version, so you'll have to fight with them to accept that it is a real problem, or capitulate and try to repro on their supported software version.

It took me 6 months to convince AWS support that there was a rare bug in network setup (not all subnets were reachable) that only hit once ever 500 - 1000 instance launches. They finally admitted that it was some sort of rare convergence problem in their network stack and that they are not monitoring for such problems so it won't recur.

At least when you own the hardware, you have full visibility into the entire stack, and while you can sitll have different teams pointing the finger at each other, they all work for the same company so management can step in and tell them to stop pointing fingers and work together to find the solution.

I hope that they use these cost savings to offer..

[supremebob]

More storage with their free tier! Seriously, guys, 2GB doesn't cut it anymore. Your competitors like Google Drive and Microsoft Onedrive are offering five times more storage for their free tier customers.