What Do You Do When the Cloud Shuts Down?

jbrodkin writes "Can you trust your data to the cloud? For users of an online storage service called The Linkup, formerly known as MediaMax, the answer turned out to be a resounding 'no.' The Linkup shut down on Aug. 8 after losing access to as much as 45% of its customers' data. 'When we looked at some individual accounts, some people didn't have any files, and some people had all their files,' The Linkup CeO Steve Iverson admits. None of the affected users will get their lost data back. Iverson called it a 'worst-case scenario.'"

Comments

Backups, backups, backups!

[morgan_greywolf]

Like anything else, including local technology, the key is to create a backup strategy. The cloud creates special problems for performing and managing backukps, so you need to understand your chosen compute or storage cluster provider's options, as well as other options specific for your application in regards to backups.

Re:Backups, backups, backups!

[pha7boy]

Redundancy, redundancy, redundancy. One backup to rule them all is not going to work. And for mission critical files would have to be backed up several times.

Re:Backups, backups, backups!

[morgan_greywolf]

No kidding. Why do you think I said 'backups' three times in the subject line? ;) That's what I mean by a 'backup strategy' -- backup strategies, which are sometimes called 'disaster recovery plans', though that's really a bigger plan that includes a backup strategy, include making multiple redundant backups, offsite storage of backups, considerations for multiple different media, etc. There are several 'best practices', but the best strategy is going to be different for each company or department and often even for each application.

The best thing to do is to examine what kind(s) of data there is in the set, how large that data set is, how often that data gets updated, how often it needs to be accessed, and what are the potential costs for losing a day's, week's, month's, year's etc. worth of that data. That will point you in the direction as to frequency of backups, types of backups, etc.

Offsite backups are essential for any data requiring backup.

Re:Backups, backups, backups!

[NorbrookC]

There's now the assumption (and we all know what assume means) that if it's "in the cloud," the data is safe or backed up somewhere. Servers fail. Backups fail. Software glitches happen. Disasters - natural or other - happen. Even if you're lucky and you don't lose the actual data, losing access to it is the same - and for an extended length of time, it can be expensive.

No matter how much we preach to the choir, it seems that most people simply don't get the message.

Re:Backups, backups, backups!

[Ihmhi]

Redundancy, redundancy, off-site redundancy

There, fixed that for you. Backups aren't worth a damn if the building is blown up.

Hm, there seems to be a pizza van outside my residence...

Re:Backups, backups, backups!

[daeg]

Don't forget to have a RESTORE strategy in place, too, and one that can be executed by others. Redundant backups don't do any good if you don't know how to restore from them, and know approximately how long it will take to restore.

We set up a test system identical to a few of our servers and had non-IT people execute the restoration plan for the core applications/data our business needs. There were a few flaws in the plan but it was a GREAT learning tool.

Re:Backups, backups, backups!

[laejoh]

Yeah, redundancy is the reason why I still have porn magazines hidden underneat the mattress! One backup to rule them all, and several magazines for the mission critical porn (don't ask!)

Re:Backups, backups, backups!

[grassy_knoll]

"No matter how much we preach to the choir, it seems that most managers simply don't want to pay for it."

Fixed that for you.

Re:Backups, backups, backups!

[westlake]

I have a script that backs up all my files from our servers in WTC1 to our servers in WTC2. What are the odds we could lose both sets of servers?
.

setting asside any questions of bad taste....

there is a real argument to be made here for "security through obscurity."

for choosing the small town industrial site that the locals haven't given a thought to in thirty-five years.

Not a new problem!

[EastCoastSurfer]

What do you do when your local computer shuts down? How about a server on your company intranet? The cloud is no different. Backups are your friend!

Re:Not a new problem!

[jgtg32a]

Kinda funny when you think about it, the backups are stored locally and the working copies are stored far away.

Re:Not a new problem!

[EastCoastSurfer]

Which leads me to this question...why is the cloud not doing backups? The cloud provider should be backing up the data within the cloud. I had assumed (wrongfully it turns out) that one of the benefits of using a cloud was that your data was backed up in some distributed fashion. It turns out that doesn't seem to be the case.

Re:Not a new problem!

[jollyreaper]

Which leads me to this question...why is the cloud not doing backups? The cloud provider should be backing up the data within the cloud. I had assumed (wrongfully it turns out) that one of the benefits of using a cloud was that your data was backed up in some distributed fashion. It turns out that doesn't seem to be the case.

That is what's happening, when the cloud is working properly. Google has a zillion servers for running Docs. Any one server dies, I don't even know about it because another steps in seamlessly.

But what happens if the whole damn cloud dies? What happens if Google goes tits up, changes terms of service, whatever?

At least for Docs, I still have the local viewer and can export from there.

I'm thinking that the best concept for cloud computing in practical terms would be the route Google is going, a mix between smart and dumb terminals. The cloud is there to host the apps but if something breaks, work can still be performed on the local. Right now, if I lose internet access I can still work in my Docs but can only view on the spreadsheets. If these were shared docs, there's still sync issues. When everyone is online, individual edits are synced every few seconds so it's difficult to screw something up. With offline edits, two different people can edit the same segment and we end up with 'last change wins' rules. This is not so bad if it's not a shared document.

Where this could become more problematic is if we're talking about database-heavy apps like say a CRM or accounting app. Naturally, it would be evil to try to cache everything locally. It would take smart programming to only cache what a person typically deals with. ACT had a laptop docking model where the laptop user could check out certain accounts which remained locked on the server and when the sales rep returned to the office, he could sync it back up. Even at that, ACT databases weren't ugly huge. Serious enterprise databases can get very large.

Well, I suppose these are problems that businesses have been struggling with for ages. It sounds stupid to say you can't do any work because the internet went out but 20 years ago people were using dumb terminal apps over frame relay and they couldn't do anything when their line went down.

Backup, Storage

[Cytlid]

I can't believe this article. The number of places you store your data is directly related to the level of which it's important to you. People put all their data in once place then cry when it's gone? How is this new?

Isn't this akin to dumping all you money into one stock then whining when it tanks?

Re:Backup, Storage

[Firehed]

Isn't this akin to dumping all you money into one stock then whining when it tanks?

Sure, but that doesn't stop people from doing exactly that.

What Do You Do When the Cloud Shuts Down?

[sm62704]

Open the curtains and let the sunshine in, and water the garden.

Oh, you mean the network... what kind of fool trusts his data with someone else?

Re:What Do You Do When the Cloud Shuts Down?

[houghi]

A fool and his data are soon to be parted.

Re:What Do You Do When the Cloud Shuts Down?

[DingerX]

Maybe the same kinds of fools who trust their money with someone else.

Seriously, cloud storage is very useful as part of a backup strategy -- offsite, maintained to professional standards. It's even more useful for geographically-disperse projects (or when I need to get at my files on the move).

But running a company that provides this sort of service is like running a bank. It's too bad Nirvanix thinks that this isn't their problem. Even if it's a screwup by an administer from the part of the old company that's now MediaMax/The Linkup, they are associated intimately with this loss. Making statements like "It's not our fault; Barney disassociated the files. That would never happen here," is just stupid. Even worse is "We have the data, but can't get at them, because to do so requires our client's front end." Guess what? It did happen here and you do have the data. It's charming that you guys managed to get into a contractual dispute over other people's data. Any contracting business now knows exactly what to expect from you.

Now, what kind of fool trusts his data with clowns like these?

Re:What Do You Do When the Cloud Shuts Down?

[CastrTroy]

Insurance means nothing. Once your data is lost, it is lost. Whether or not you get money out of them in compensation for the lost data is almost non-important. I would say that anything you lost would be completely non-producable, even if you had all the money in the world. A picture of your family on vacation, can't be reproduced. You can go on another vacation, but it won't be the same vacation. Any document you have typed out, could be typed out again, but it would be different each time. Unless you are talking about lost music files, in which case, you could download them again, but that's kind of the same as having a backup. Any data that's really important isn't going to reproducible.

Re:What Do You Do When the Cloud Shuts Down?

[IceCreamGuy]

Whether or not you get money out of them in compensation for the lost data is almost non-important

Maybe for a home user, but usually for a business time actually does = money, and the two are relatively interchangeable. One day of work = $x, and insurance means a lot. And if you'd actually read TFA, you would know that one of the companies involved, Nirvanix, is a business oriented cloud-storage company.

Maybe what you meant to say was "Insurance means nothing to me."

Re:What Do You Do When the Cloud Shuts Down?

[CastrTroy]

But it's not as simple as you put it. Let's say all my data was on the cloud. And I spend a 30 days writing up some code for a client. And I charged the client a rate of $1000 a day. Based on days*cost per day, I would get $30,000. But by losing all that data, I have lost much more. Maybe there was something in the contract (there should have been), stating that if I was late, I would have to lower the final price the client paid me, and not be paid for the additional days. What about all the other clients I promised work to, who's contracts I can't fulfill because I'm stuck having to rewrite the code I lost for the first client. Maybe that client no longer wants to deal with me for future projects. Maybe that client will tell a bunch of other potential clients how unreliable I am, and I will lose future clients. There's no easy way to figure out how much a single days work is actually worth. It seems in most cases, that the best "insurance" would be to create multiple backups, rather than trust your data to "the cloud".

The critical flaw

[Nephroth]

The critical flaw of cloud computing is that you entrust your data to a third party. If you are at all concerned with privacy you will think cloud computing is a terrible idea.

Re:The critical flaw

[bencoder]

Well that's what encryption is for. Seriously, I'd much much rather have my data encrypted on a remote server than have it unencrypted on my own computer, especially if I ever want to go to the US.
In fact, even if it's encrypted locally, that means I'm even more likely to lose it at the border because if it's encrypted then surely I'm an evil terrorist come to take away all your freedoms that you enjoy, such as your protection from unreasonable search or seizure...

Re:The critical flaw

[19thNervousBreakdown]

What's preventing a service that does encryption/decryption on the client side? Other than the lack of desire from the providers I mean.

Re:The critical flaw

[phoenix.bam!]

Cloud computing is no different than any other hosting service. Shared hosting, a colo box, a virtual machine, or a cloud account are all vulnerable. So unless you have a direct line to a tier1 backbone you're going to have to put your data into someone else's hands at some point.

An old maxim:

[Tinfoil]

If you want something done right, do it yourself.

Those who would knowingly trust their data to an outside (and relatively untested) organization without having a backup in place are just asking for something like this to happen.

Oh, ya, backups are hard.

Test restore, test restore, test restore!

[gilgongo]

I don't have any way of verifying this story, but I worked with an old guy once who told me that he had been at a startup in the UK that was, by the sound of it, creating a kind of IMDB in about 1994. They had a team of researchers and a bunch of seed capital to create a large film database. Everything was ticking along for about 18 months and they had researched thousands of films.

Then one day, the database shut down and they traced it to some bad hardware. They replaced the hardware and restored the database from the previous night's backup. Nothing doing - the backup tape (he said it was DAT) was corrupt. So they tried the other one. Nada. Same corruption. So they tried the off-site one. Same thing. Turned out all the backups they had made seem to have transferred the same corruption resulting in nothing significant recoverable.

Had they tried a test restore at some point, they might have found out. As it was, a week after the crash, they shut the business down.

Which reminds me of another (maybe apocryphal) story: the head of IT as a large company was fond of organising disaster recovery practices by walking into the data centre, physically removing a (pre-ordained) server and leaving a note in its place with the words "The server crashed" written on it. The support staff (and presumably management) knew that this would happen, but not when, or which machine (or dependent services) would be affected. Interesting test I would say.