Slashdot Mirror
Mega-Uploads: The Cloud's Unspoken Hurdle
First time accepted submitter n7ytd writes "The Register has a piece today about overcoming one of the biggest challenges to migrating to cloud-based storage: how to get all that data onto the service provider's disks. With all of the enterprisey interweb solutions available, the oldest answer is still the right one: ship them your disks. Remember: 'Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.'"
Returning from a site with a tethered computer full of 80 MP 16-bit raw files from a day's worth of shooting would break most bandwidth bills if you tried uploading all these images.
Pressed if disks are accepted, the company responded that “All common database products provide a capability to extract to a common file format like .csv.”
what a professional answer. and by that i mean it didn't answer the question at all.
insensitive clod overlords obligatory xkcd car analogy russian reversals whoosh pedant fanbois ftfy in 3...2...1..PROFIT
Yeah, the bandwidth is great, but the latency SUCKS.
Second biggest challenge: trusting any of these places have the motivation to keep your data more secure than credit card companies do.
Getting around all the buzzwords
Well that's one hurdle.
The next is RECOVERY when ICE or FBI or some other 3letter agency walks in an takes your data because one tiny customer use the service for some allegedly nefarious purpose.
The key here is to use a service so big that even god himself would not dare take it down, although the Ayatollah might try. Small cloud services, even if multi-homed are a risky proposition. Even if you do manage to get all your data into them, they are not large enough to push back against any subpoena or search warrant that any misguided judge in some backwater jurisdiction may issue.
Sig Battery depleted. Reverting to safe mode.
Comment removed based on user account deletion
My last employer offered offsite backups to clients. For the initial seed, we always tried to get them to put it on an external HDD and ship it to us (or at least DVDs). The only major exceptions were clients that were also on FiOS - that was the only case where over-the-net transfer was faster than the backup-and-ship-it method for the initial seed.
I don't think that TFS's answer is necessarily the correct one. I'd really prefer to hold Ma Bell's feet to the fire concerning the fact that bandwidth(even in 'optimal' build-out areas, spare me the excuses about the boonies) has enjoyed a deeply underwhelming track record in terms of improvements in cost and quantity compared to most other aspects of contemporary computing.
Yes, never underestimate the bandwidth of a station wagon full of disks hurtling down the highway. The latency, on the other hand, leaves much to be desired, and I've heard the packet loss can be downright fatal.
--- Journals are boring; Go to my web page instead
No, the second biggest challenge is to come up with a viable exit strategy. Once you have several TB at this service provider, how will you move it out of there when the next provider has a better deal? That was one of the major big points for having a cloud in the first place, to have the freedom to move your compute requirements to a better, cheaper, faster (pick two) provider.
Even if you moved it in with a station wagon full of tapes or disks and your provider let you import it, I'm sure your provider will not be so helpful when you need to move it back out.
Blatant plug: Perhaps Actifio (www.actifio.com) can fix this for you, by replicating your data in, and also back out of production systems in deduped and compressed format.
To Terminate, or not to Terminate, that's the question - SCSIROB
... is getting it all back OFF again when you want to switch service providers.
The one thing you want never to happen is that you get locked in to a single cloud service. They might go bust, they might become uncompetitive. They may become politically "unfriendly" or tainted with customers you have no desire to be associated with - or any of a number of other reasons to say "adios".
Just like with disaster planning, all the processes and procedures, agreements and SLAs are worthless until you've actually PERFORMED the operation and done so without a major service interruption. How many cloud users have gone that far - and how many are locked in but don't know it?
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
You you always use a UDP solution such as Aspera. Fast transfer speeds, bandwidth management and they have a specific AWS implimentation.
Other options to look at include Smartjog, whose new Bolt product looks quite interesting, Riverbed's Steelhead product, Filecatalyst and Signiant.
There are many solutions around now to deal with large file transfers for both small and large business. Most of them use UDP instead of TCP/IP, with Checksums to ensure all data is reliable delivered. Even with just 1Mbps upload speeds, something like one of the above named products will be advantageous. I've worked in the media industry for a number of years, and this type of thing is being used in Film and Television all the time. Of course, there are still tapes being shipped around, but in emerging markets, such as Russia for instance, the file transfer really beats a tape being stuck in customs for weeks or months.
Brought to you by the author of such childrens' classics as "Some Kittens can Fly!" and "All Dogs go to Hell."
How did you manage to fix armed FBI storming your servers located in another country problem?
I am dealing with this as well, albeit on a different scale. About a year ago, the powers that be decided that they were going to develop a private cloud for the company. Nobody really considered how to migrate 500+TB of data from three separate sites into the new cloud. We are doing a mixture of over the wire replication (for sites with 100Mb+ of bandwidth), physical replication (using NAS devices and tape), and synchronization using DoubleTake for the SQL data and Vice Versa Pro for file system data. It is a massive undertaking, made even more difficult by the fact that we are working with production systems with locked in SLAs that need be maintained.
For the average person, and even most enterprises, I honestly believe the best way to get into "the cloud" is by following a well planned out, phased approach. The first phase should be using the cloud as a DR target. Only when both sides of the equation are balanced and able to operate independently of each other can you consider doing away with one and moving to the other.
That is just one of many of the hurdles.
Really, these problems are problems because most 'cloud' shit is done wrong.
It's a bit of a worn out record here on Slashdot, but anyone or any company which is fully dependent upon The Cloud for business continuity is a fool.
* First off, there is no such thing as 'utility computing', and probably never will be due to the volatile nature of storage and its ongoing cost of maintenance.
* Second, if you do not maintain primary physical control of something, to the best of your ability, you do not control it.
* For primary IT infrastructure, it will cost more to do "Cloud" than local. If you can afford 2-3 servers a year, but not much more, and a nominal IT operations budget, chances are you should have an in-house "cloud" with off-site replication.
* Bandwidth costs both ways will kill you, as will latency in many cases, will kill Cloud functionality.
At this point, I still strongly recommend against public Clouding your systems unless they are:
a) (very!) low volume with use-based billing. This only makes sense for a low-volume public-facing site where you don't already have IT infrastructure (on a cost basis)
b) off-site 'hot' replication. You've got your inside 'private Cloud' which replicates to off-site systems. (Cloud is basically just colocated virtualization, after all.)
c) Other geographic/distribution requirements (eg. multisite organization with none serving as a good central hub). In this case, colocation of your own equipment makes more sense in many regards.
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
In canada, unless you need low latency, the internet is about the most expensive method you could possibly use to transfer data. source
http://aws.amazon.com/importexport/
http://awsimportexport.s3.amazonaws.com/aws-import-export-calculator.html
It's not rocket science. Yes, shipping drives is the cheapest, fastest option for a lot of people.
YMMV, speaking for myself, not my employer, etc. etc.
-Isaac
I am not a lawyer, and this is not legal advice. For Entertainment Purposes Only.
African or European?
Namely, the increased risk that your data will become collateral damage in the War On Piracy.