Mega-Uploads: The Cloud's Unspoken Hurdle

First time accepted submitter n7ytd writes "The Register has a piece today about overcoming one of the biggest challenges to migrating to cloud-based storage: how to get all that data onto the service provider's disks. With all of the enterprisey interweb solutions available, the oldest answer is still the right one: ship them your disks. Remember: 'Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.'"

Comments

Pro photography is a huge problem

[Anonymous Coward]

Returning from a site with a tethered computer full of 80 MP 16-bit raw files from a day's worth of shooting would break most bandwidth bills if you tried uploading all these images.

Re:Pro photography is a huge problem

[HapSlappy_2222]

Just because you take pro pictures at 80MP doesn't mean your business has an extra $10,000 per year laying around for a business grade gigabit pipe; as the sole employee of my company, that'd mean I'm paying myself $20,000 a year instead of $30,000 (TBH, I'm lucky to be in the black, period, with only two years in). I store my images at my studio, back them up daily to a removable disk, and bring in a 3rd removable to copy them over once per week. All told? $500 for the 2 drives and a striped array on my studio PC. The self-storage backup technique works well for me.

Realistically, though, if I want to I can just upload them all to home or a cloud storage in batches overnight, the same way I download 10 gigabyte files at home. It's just plain easier to cart em around, though.

Re:Pro photography is a huge problem

[rthille]

The tiny town of Sebastopol CA, population ~7800 has gigabit fiber to the (some) doorstep for $69/month.

Station Wagon Full of Tapes

[viking099]

Remember: 'Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.'"

Yeah, the bandwidth is great, but the latency SUCKS.

Second biggest challenge

[WOOFYGOOFY]

Second biggest challenge: trusting any of these places have the motivation to keep your data more secure than credit card companies do.

Re:Second biggest challenge

[gestalt_n_pepper]

No, that's third. The second biggest challenge is believing that those fine hosting companies with servers hosted in lower Slobbovia won't have a few entrepreneurial employees who will *actively* be searching your data for all that is monetizable.

Re:The real hurdle

[icebike]

Getting around all the buzzwords

Well that's one hurdle.
The next is RECOVERY when ICE or FBI or some other 3letter agency walks in an takes your data because one tiny customer use the service for some allegedly nefarious purpose.

The key here is to use a service so big that even god himself would not dare take it down, although the Ayatollah might try. Small cloud services, even if multi-homed are a risky proposition. Even if you do manage to get all your data into them, they are not large enough to push back against any subpoena or search warrant that any misguided judge in some backwater jurisdiction may issue.

Backups

[SJHillman]

My last employer offered offsite backups to clients. For the initial seed, we always tried to get them to put it on an external HDD and ship it to us (or at least DVDs). The only major exceptions were clients that were also on FiOS - that was the only case where over-the-net transfer was faster than the backup-and-ship-it method for the initial seed.

Bandwidth of a Station Wagon

[Surazal]

Yes, never underestimate the bandwidth of a station wagon full of disks hurtling down the highway. The latency, on the other hand, leaves much to be desired, and I've heard the packet loss can be downright fatal.

Re:Bandwidth of a station wagon

[Firethorn]

Indeed. He also ignored the core reason for having said bandwidth - you have X amount of data to move in Y time (at under Z cost); what's the best way to do so?

As such, a 'packet' on the freeway system is rather expensive, so you don't want to be putting multiple station wagons on the system if you don't have to. Figure the driver costs $20/hour, the vehicle itself $.50/mile(gas, maintenance, insurance, tolls, etc...), and you're looking at 300 miles in 10 hours. For a single packet you're looking at $350 for that single 'packet'. If a single station wagon doesn't do it, perhaps a cargo van would, which doubles the capacity of the packet while only raising the cost $50, to $400. Still not good enough? Upgrade to a 'package van' like UPS/Fedex trucks. Next step would be a Semi.

In any case, I'd say that you could fit 25TB into a motorcycle today - 3 TB drives are fairly common now, and I can fit 10 into my saddlebags easily. Heck, I can get 1.5TB native tapes, about the same size as a HD. Padding it's dimensions up, it's 11 x 11 x 3 cm = 363 cm^3, or 2,755 per cubic meter.

A 2008-11 Dodge Grand Caravan Cargo van - 143.8 cubic feet = 4.07 cubic meters, giving me room for 11k 1.5TB tapes. 16.5k TB, in 10 hours, if I have a single cargo van. Ouch. Disregarding media cost, that's ~$400.

Do this daily, we're looking at 1.5 terrabits per second. Don't know of any connections that fast.
Monthly, we're down to ~50 gigabit (rounding down). I can guarantee that a 50 gigabit connection will cost more than $400.
Annually, it's 'only' 4 gigabit, and I pay more than $100/month for my megabit class connection, which ISN'T utilized 100%, unlike my calc.

You don't normally need to figure out the bandwidth of the freeway because:
1. Generally 1 vehicle 'packet' is sufficient, and due to the high marginal cost per said vehicle, you normally only want to send one.
2. The roads are used for more than data shipment, which would be like trying to figure out how much bandwidth you have available for VOIP by looking at total circuit bandwidth.

Don't need to ship that much? You should be able to ship about 30 of them for $60, second day air. That's 45TB, or about 140 Mbit of 100% saturated traffic for a month. BTW, during my calcs for paying fedex to ship them, I think that weight might actually be enough of an issue to increase gasoline consumption - but I think I've established that even $800 would be cheap if you need to ship that ridiculous of an amount of data.

A problem bigger than getting your data on ...

[petes_PoV]

... is getting it all back OFF again when you want to switch service providers.

The one thing you want never to happen is that you get locked in to a single cloud service. They might go bust, they might become uncompetitive. They may become politically "unfriendly" or tainted with customers you have no desire to be associated with - or any of a number of other reasons to say "adios".

Just like with disaster planning, all the processes and procedures, agreements and SLAs are worthless until you've actually PERFORMED the operation and done so without a major service interruption. How many cloud users have gone that far - and how many are locked in but don't know it?

But then...

[TemperedAlchemist]

How did you manage to fix armed FBI storming your servers located in another country problem?

Re:The real hurdle

[CAIMLAS]

That is just one of many of the hurdles.

Really, these problems are problems because most 'cloud' shit is done wrong.

It's a bit of a worn out record here on Slashdot, but anyone or any company which is fully dependent upon The Cloud for business continuity is a fool.

* First off, there is no such thing as 'utility computing', and probably never will be due to the volatile nature of storage and its ongoing cost of maintenance.
* Second, if you do not maintain primary physical control of something, to the best of your ability, you do not control it.
* For primary IT infrastructure, it will cost more to do "Cloud" than local. If you can afford 2-3 servers a year, but not much more, and a nominal IT operations budget, chances are you should have an in-house "cloud" with off-site replication.
* Bandwidth costs both ways will kill you, as will latency in many cases, will kill Cloud functionality.

At this point, I still strongly recommend against public Clouding your systems unless they are:

a) (very!) low volume with use-based billing. This only makes sense for a low-volume public-facing site where you don't already have IT infrastructure (on a cost basis)
b) off-site 'hot' replication. You've got your inside 'private Cloud' which replicates to off-site systems. (Cloud is basically just colocated virtualization, after all.)
c) Other geographic/distribution requirements (eg. multisite organization with none serving as a good central hub). In this case, colocation of your own equipment makes more sense in many regards.