Slashdot Mirror


What Do You Do When the Cloud Shuts Down?

jbrodkin writes "Can you trust your data to the cloud? For users of an online storage service called The Linkup, formerly known as MediaMax, the answer turned out to be a resounding 'no.' The Linkup shut down on Aug. 8 after losing access to as much as 45% of its customers' data. 'When we looked at some individual accounts, some people didn't have any files, and some people had all their files,' The Linkup CeO Steve Iverson admits. None of the affected users will get their lost data back. Iverson called it a 'worst-case scenario.'"

48 of 203 comments (clear)

  1. Backups, backups, backups! by morgan_greywolf · · Score: 5, Insightful

    Like anything else, including local technology, the key is to create a backup strategy. The cloud creates special problems for performing and managing backukps, so you need to understand your chosen compute or storage cluster provider's options, as well as other options specific for your application in regards to backups.

    1. Re:Backups, backups, backups! by pha7boy · · Score: 3, Insightful

      Redundancy, redundancy, redundancy. One backup to rule them all is not going to work. And for mission critical files would have to be backed up several times.

      --
      -- All this knowledge is giving me a raging brainer.
    2. Re:Backups, backups, backups! by morgan_greywolf · · Score: 5, Insightful

      No kidding. Why do you think I said 'backups' three times in the subject line? ;) That's what I mean by a 'backup strategy' -- backup strategies, which are sometimes called 'disaster recovery plans', though that's really a bigger plan that includes a backup strategy, include making multiple redundant backups, offsite storage of backups, considerations for multiple different media, etc. There are several 'best practices', but the best strategy is going to be different for each company or department and often even for each application.

      The best thing to do is to examine what kind(s) of data there is in the set, how large that data set is, how often that data gets updated, how often it needs to be accessed, and what are the potential costs for losing a day's, week's, month's, year's etc. worth of that data. That will point you in the direction as to frequency of backups, types of backups, etc.

      Offsite backups are essential for any data requiring backup.

    3. Re:Backups, backups, backups! by Firehed · · Score: 2, Informative

      True, but the cloud (at least in theory) also gives you ubiquitous access to your data from any location - when it's online, of course. I think it's best to treat it as the centralized synchronization point that you work from day-to-day (think: email, calendar) so that it's consistent across your devices, but have at least one system that YOU control periodically backing up that data.

      The issues with services like Gmail and Amazon S3 tend not to be with hardware failures, but with software problems. Recent S3 outages as well as yesterday's brief Gmail outage weren't caused by a load balancer exploding, but by something going screwy in software. Theoretically no data will be erased if they've got permissions set right, but that doesn't mean you're important enough to get a human working on the database to fix YOUR account.

      --
      How are sites slashdotted when nobody reads TFAs?
    4. Re:Backups, backups, backups! by NorbrookC · · Score: 2, Insightful

      There's now the assumption (and we all know what assume means) that if it's "in the cloud," the data is safe or backed up somewhere. Servers fail. Backups fail. Software glitches happen. Disasters - natural or other - happen. Even if you're lucky and you don't lose the actual data, losing access to it is the same - and for an extended length of time, it can be expensive.

      No matter how much we preach to the choir, it seems that most people simply don't get the message.

    5. Re:Backups, backups, backups! by Ihmhi · · Score: 2, Insightful

      Redundancy, redundancy, off-site redundancy

      There, fixed that for you. Backups aren't worth a damn if the building is blown up.

      Hm, there seems to be a pizza van outside my residence...

    6. Re:Backups, backups, backups! by morgan_greywolf · · Score: 2, Funny

      There's now the assumption (and we all know what assume means) that if it's "in the cloud," the data is safe or backed up somewhere

      These must be the same people that assume that because someone has a fancy degree, they know what they're talking about.

    7. Re:Backups, backups, backups! by daeg · · Score: 4, Insightful

      Don't forget to have a RESTORE strategy in place, too, and one that can be executed by others. Redundant backups don't do any good if you don't know how to restore from them, and know approximately how long it will take to restore.

      We set up a test system identical to a few of our servers and had non-IT people execute the restoration plan for the core applications/data our business needs. There were a few flaws in the plan but it was a GREAT learning tool.

    8. Re:Backups, backups, backups! by jollyreaper · · Score: 4, Funny

      There, fixed that for you. Backups aren't worth a damn if the building is blown up.

      Yeah, I already thought of that. *smug* I have a script that backs up all my files from our servers in WTC1 to our servers in WTC2. What are the odds we could lose both sets of servers?

      --
      Kwisatz Haderach
      Sell the spice to CHOAM
      This Mahdi took Shaddam's Throne
    9. Re:Backups, backups, backups! by laejoh · · Score: 2, Insightful

      Yeah, redundancy is the reason why I still have porn magazines hidden underneat the mattress! One backup to rule them all, and several magazines for the mission critical porn (don't ask!)

    10. Re:Backups, backups, backups! by lawpoop · · Score: 3, Interesting

      This is why I always thought that cloud computing based on servers would be disastrous. What if the server goes down? Well, here's a case in point. You lose everything.

      I proposed an idea like a P2P backup. Say you have some 20 GB you want to back up. You make 20 GB available on your system, and fire up a P2P backup program. You partner with people who want to backup also, trade backup space, and voila! You have a distributed backup system. It's all encrypted, so you can't get into other people's stuff on your system, and vice-versa. Periodically, the app checks to make sure that all your backup partners are available. If not, it starts negotiating a backup with a new partner.

      Of course, you don't want to lose your stuff to a single host going down, you would have a ratio of 3:1 or 4:1 to make sure that you have high availability.

      --
      Computers are useless. They can only give you answers.
      -- Pablo Picasso
    11. Re:Backups, backups, backups! by nmg196 · · Score: 3, Informative

      That idea is already quite prevalent and many programs such as BuddyBackup already use this idea.

    12. Re:Backups, backups, backups! by grassy_knoll · · Score: 4, Insightful

      "No matter how much we preach to the choir, it seems that most managers simply don't want to pay for it."

      Fixed that for you.

    13. Re:Backups, backups, backups! by bjk002 · · Score: 5, Funny

      "No kidding. Why do you think I said 'backups' three times in the subject line?"

      I think he was just backing up your statement!

      --
      Opinion:=TMyOpinion.Create(Me);
    14. Re:Backups, backups, backups! by mrdoogee · · Score: 2, Funny

      Personally, I back up my data in an inverse parallel universe. Several parallel universes actually. One is a cowboy universe!

    15. Re:Backups, backups, backups! by rkanodia · · Score: 3, Funny

      I used to back up my data in the mustache-and-goatee universe, but it kept coming back with the Evil bit set.

    16. Re:Backups, backups, backups! by westlake · · Score: 2, Insightful
      I have a script that backs up all my files from our servers in WTC1 to our servers in WTC2. What are the odds we could lose both sets of servers?
      .

      setting asside any questions of bad taste....

      there is a real argument to be made here for "security through obscurity."

      for choosing the small town industrial site that the locals haven't given a thought to in thirty-five years.

  2. Not a new problem! by EastCoastSurfer · · Score: 4, Insightful

    What do you do when your local computer shuts down? How about a server on your company intranet? The cloud is no different. Backups are your friend!

    1. Re:Not a new problem! by jgtg32a · · Score: 5, Insightful

      Kinda funny when you think about it, the backups are stored locally and the working copies are stored far away.

    2. Re:Not a new problem! by Tim+C · · Score: 2, Interesting

      What do you do when your local computer shuts down? How about a server on your company intranet?

      Well with the former I can pull the hard drive and shove it in a new machine and be at least trying to recover my data inside of an hour. With the latter, the systems team could be doing the equivalent inside of a day (as the servers don't tend to be in the office).

      If my remote document storage/app server/whatever goes down, even transiently, there's nothing I can do until it comes back up (other than hope that it does come back up).

      So yes, backups are your friend, but the situation isn't quite the same.

    3. Re:Not a new problem! by EastCoastSurfer · · Score: 2, Insightful

      Which leads me to this question...why is the cloud not doing backups? The cloud provider should be backing up the data within the cloud. I had assumed (wrongfully it turns out) that one of the benefits of using a cloud was that your data was backed up in some distributed fashion. It turns out that doesn't seem to be the case.

    4. Re:Not a new problem! by jollyreaper · · Score: 2, Insightful

      Which leads me to this question...why is the cloud not doing backups? The cloud provider should be backing up the data within the cloud. I had assumed (wrongfully it turns out) that one of the benefits of using a cloud was that your data was backed up in some distributed fashion. It turns out that doesn't seem to be the case.

      That is what's happening, when the cloud is working properly. Google has a zillion servers for running Docs. Any one server dies, I don't even know about it because another steps in seamlessly.

      But what happens if the whole damn cloud dies? What happens if Google goes tits up, changes terms of service, whatever?

      At least for Docs, I still have the local viewer and can export from there.

      I'm thinking that the best concept for cloud computing in practical terms would be the route Google is going, a mix between smart and dumb terminals. The cloud is there to host the apps but if something breaks, work can still be performed on the local. Right now, if I lose internet access I can still work in my Docs but can only view on the spreadsheets. If these were shared docs, there's still sync issues. When everyone is online, individual edits are synced every few seconds so it's difficult to screw something up. With offline edits, two different people can edit the same segment and we end up with 'last change wins' rules. This is not so bad if it's not a shared document.

      Where this could become more problematic is if we're talking about database-heavy apps like say a CRM or accounting app. Naturally, it would be evil to try to cache everything locally. It would take smart programming to only cache what a person typically deals with. ACT had a laptop docking model where the laptop user could check out certain accounts which remained locked on the server and when the sales rep returned to the office, he could sync it back up. Even at that, ACT databases weren't ugly huge. Serious enterprise databases can get very large.

      Well, I suppose these are problems that businesses have been struggling with for ages. It sounds stupid to say you can't do any work because the internet went out but 20 years ago people were using dumb terminal apps over frame relay and they couldn't do anything when their line went down.

      --
      Kwisatz Haderach
      Sell the spice to CHOAM
      This Mahdi took Shaddam's Throne
  3. Backup, Storage by Cytlid · · Score: 5, Insightful

    I can't believe this article. The number of places you store your data is directly related to the level of which it's important to you. People put all their data in once place then cry when it's gone? How is this new?

    Isn't this akin to dumping all you money into one stock then whining when it tanks?

    --
    FLR
    1. Re:Backup, Storage by Firehed · · Score: 3, Insightful

      Isn't this akin to dumping all you money into one stock then whining when it tanks?

      Sure, but that doesn't stop people from doing exactly that.

      --
      How are sites slashdotted when nobody reads TFAs?
  4. What Do You Do When the Cloud Shuts Down? by sm62704 · · Score: 4, Insightful

    Open the curtains and let the sunshine in, and water the garden.

    Oh, you mean the network... what kind of fool trusts his data with someone else?

    --
    mcgrew's razor: Never attribute to stupidity that which can be explained by greedy self-interest
    1. Re:What Do You Do When the Cloud Shuts Down? by houghi · · Score: 2, Insightful

      A fool and his data are soon to be parted.

      --
      Don't fight for your country, if your country does not fight for you.
    2. Re:What Do You Do When the Cloud Shuts Down? by dnwq · · Score: 2, Funny

      Oh, you mean the network... what kind of fool trusts his data with someone else?

      I build my own hard drives and power supply too!

    3. Re:What Do You Do When the Cloud Shuts Down? by DingerX · · Score: 2, Insightful

      Maybe the same kinds of fools who trust their money with someone else.

      Seriously, cloud storage is very useful as part of a backup strategy -- offsite, maintained to professional standards. It's even more useful for geographically-disperse projects (or when I need to get at my files on the move).

      But running a company that provides this sort of service is like running a bank. It's too bad Nirvanix thinks that this isn't their problem. Even if it's a screwup by an administer from the part of the old company that's now MediaMax/The Linkup, they are associated intimately with this loss. Making statements like "It's not our fault; Barney disassociated the files. That would never happen here," is just stupid. Even worse is "We have the data, but can't get at them, because to do so requires our client's front end." Guess what? It did happen here and you do have the data. It's charming that you guys managed to get into a contractual dispute over other people's data. Any contracting business now knows exactly what to expect from you.

      Now, what kind of fool trusts his data with clowns like these?

    4. Re:What Do You Do When the Cloud Shuts Down? by phillymjs · · Score: 3, Funny

      You don't generate your own electricity, too? Slacker!

    5. Re:What Do You Do When the Cloud Shuts Down? by CastrTroy · · Score: 5, Insightful

      Insurance means nothing. Once your data is lost, it is lost. Whether or not you get money out of them in compensation for the lost data is almost non-important. I would say that anything you lost would be completely non-producable, even if you had all the money in the world. A picture of your family on vacation, can't be reproduced. You can go on another vacation, but it won't be the same vacation. Any document you have typed out, could be typed out again, but it would be different each time. Unless you are talking about lost music files, in which case, you could download them again, but that's kind of the same as having a backup. Any data that's really important isn't going to reproducible.

      --

      Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
    6. Re:What Do You Do When the Cloud Shuts Down? by IceCreamGuy · · Score: 2, Insightful

      Whether or not you get money out of them in compensation for the lost data is almost non-important

      Maybe for a home user, but usually for a business time actually does = money, and the two are relatively interchangeable. One day of work = $x, and insurance means a lot. And if you'd actually read TFA, you would know that one of the companies involved, Nirvanix, is a business oriented cloud-storage company.

      Maybe what you meant to say was "Insurance means nothing to me."

    7. Re:What Do You Do When the Cloud Shuts Down? by CastrTroy · · Score: 2, Insightful

      But it's not as simple as you put it. Let's say all my data was on the cloud. And I spend a 30 days writing up some code for a client. And I charged the client a rate of $1000 a day. Based on days*cost per day, I would get $30,000. But by losing all that data, I have lost much more. Maybe there was something in the contract (there should have been), stating that if I was late, I would have to lower the final price the client paid me, and not be paid for the additional days. What about all the other clients I promised work to, who's contracts I can't fulfill because I'm stuck having to rewrite the code I lost for the first client. Maybe that client no longer wants to deal with me for future projects. Maybe that client will tell a bunch of other potential clients how unreliable I am, and I will lose future clients. There's no easy way to figure out how much a single days work is actually worth. It seems in most cases, that the best "insurance" would be to create multiple backups, rather than trust your data to "the cloud".

      --

      Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
  5. The critical flaw by Nephroth · · Score: 5, Insightful

    The critical flaw of cloud computing is that you entrust your data to a third party. If you are at all concerned with privacy you will think cloud computing is a terrible idea.

    --
    Our greatest enemy is neither a single man, nor is it a nation, it is, as it has always been, our own greed.
    1. Re:The critical flaw by bencoder · · Score: 3, Insightful

      Well that's what encryption is for. Seriously, I'd much much rather have my data encrypted on a remote server than have it unencrypted on my own computer, especially if I ever want to go to the US.
      In fact, even if it's encrypted locally, that means I'm even more likely to lose it at the border because if it's encrypted then surely I'm an evil terrorist come to take away all your freedoms that you enjoy, such as your protection from unreasonable search or seizure...

    2. Re:The critical flaw by 19thNervousBreakdown · · Score: 2, Insightful

      What's preventing a service that does encryption/decryption on the client side? Other than the lack of desire from the providers I mean.

      --
      <xml><I><am><so><damn>Web 2.0</damn></so></am></I></xml>
    3. Re:The critical flaw by Kadin2048 · · Score: 2, Interesting

      What's preventing a service that does encryption/decryption on the client side? Other than the lack of desire from the providers I mean.

      I think there are several issues, and lack of interest from providers is actually among the least of them. Lack of interest from users is probably the biggest issue. Many people just don't care that much about the privacy of their data -- they either honestly don't care who sees their stuff, or don't care enough to be willing to expend any effort or time preventing it. Adding a well-designed encryption feature to a backup service would add complexity and expense, and if people aren't willing to pay for that (i.e. they don't care), it's not going to get added.

      Some enterprise backup systems do offer encryption (and also offer non-cloud-based backup, for companies that want to keep everything on-site) so it's not as though it's never been done. You just don't see it on the consumer level that often, because consumers don't care enough about privacy to pay for it.

      Also, a well-designed encryption system -- where the data was effectively lost if the user misplaced or lost their decryption key -- would probably lead to a lot of customer-service problems. Inevitably, users would upload data, not make a backup of the key (or make a backup and then lose it, or not store it off-site, or whatever), and then get upset when they couldn't recover their data from the backup service that they paid for. I've dealt with this sort of thing personally before; many users just don't get encryption. They find it inconceivable that you, as the god-like administrator, can't just open up an encrypted file on command when they lose their password. (I've had people literally accuse me of plotting against them or being insubordinate for not opening their encrypted files for them.)

      So given that it would add complexity to the implementation to do right, isn't something that people are likely to pay extra for, and is likely to cause a lot of problems and expense down the road, it's not surprising that most online backup services either don't offer encryption or don't turn it on by default if they do. It's easy for knowledgeable users to add encryption to files before uploading them; just letting them do that is a lot easier than trying to explain to Aunt Millie why her vacation photos are gone because she wrote down her password on a piece of paper that was lost in the fire that destroyed her computer.

      --
      "Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
    4. Re:The critical flaw by phoenix.bam! · · Score: 2, Insightful

      Cloud computing is no different than any other hosting service. Shared hosting, a colo box, a virtual machine, or a cloud account are all vulnerable. So unless you have a direct line to a tier1 backbone you're going to have to put your data into someone else's hands at some point.

  6. Re:Well DUH! by spoonist · · Score: 2, Funny

    No, no, no, you have it all wrong.

    The first rule of backups is: "You do not talk about backups."

  7. OK, so I'm a dinosaur, but... by FlyByPC · · Score: 3, Funny

    ...they can have my local computing and storage capabilities when they pry them from my cold, dead hands. Google is great for looking things up -- and webmail accounts are great for portability -- but the old saying applies: If you want something done (backed up, available), do it yourself. Much more secure that way, too.

    Besides, with Remote Desktop, FOUSs*, and continuous 'Net connections, it's pretty easy to take it with you.


    * (8GB on a microSDHC the size of my fingernail is a Flashdrive Of Unusual Size in my book!)

    --
    Paleotechnologist and connoisseur of pretty shiny things.
  8. An old maxim: by Tinfoil · · Score: 2, Insightful

    If you want something done right, do it yourself.

    Those who would knowingly trust their data to an outside (and relatively untested) organization without having a backup in place are just asking for something like this to happen.

    Oh, ya, backups are hard.

  9. Sell me data insurance by cohomology · · Score: 2, Interesting

    Every year, I read the terms of service of a bunch of online backup services, but I have not found one that gives the provider any incentive to be careful. They say they have *no liability of any kind*. Why should I trust them?

    I will cheerfully pay to insure access to my data, but nobody offers me insurance.

    --
    Don't mess with The Phone Company. Piss them off and you'll be using two tin cans and a piece of string.
  10. Same happened to me. by anomnomnomymous · · Score: 4, Funny

    I had this overly insightful comment... but it all got lost when I submitted it.

    And now look what I'm left with!

    --
    When you shoot a mime, do you use a silencer?
  11. This is why backups are important by Jason+Levine · · Score: 2, Interesting

    A few years ago, I had my websites hosted at this one company, Digi-Wave. They were great for a few years, but suddenly their servers were down. For a week. Yes, I said a week. The servers came up again briefly before going down again, but in that brief span, I managed to backup my database and files. When I called their support line, I was told that their servers were infected with Code Red (IIRC, I know it was one of those IIS worms). I knew this was a bogus answer because the fix to Code Red infection was: 1) disconnect the machine from the 'Net, 2) reboot it, 3) apply the patch (possibly rebooting again), 4) reconnect to the 'Net. It shouldn't have taken them over a week to fix this.

    Then they stopped answering support calls and their phone's inbox filled up until it stopped accepting recordings. By this time, I contacted my credit card company to get my money back and had made arrangements with another hosting provider. I was lucky to have retained my data. Many were not so lucky. And to add insult to injury, after Digi-Wave folded, another hosting company arose with a different name but the same contact information.

    The moral of this story is to always backup. Because you never know when the cloud, your webhost, or even your personally owned and run server will go south and take your data with it.

    --
    My sci-fi novel, Ghost Thief, is now available from Amazon.com.
  12. Link to the article on ONE PAGE by Dekortage · · Score: 3, Funny

    This version may be easier to read.

    Bottom line: The Linkup is blaming Nirvanix (a third-party service provider) which is, of course, blaming The Linkup. FTA:

    Nirvanix says it has not deleted any customer data, and promises that its Storage Delivery Network is immune to the problem that plagued The Linkup. At The Linkup, a "system administrator ran a script that misidentified active account data and disassociated physical files from their owners," Nirvanix says. "This led to files being marked offline in the old Streamload/MediaMax file system when they shouldn't have been." Iverson, meanwhile, claims it was a Nirvanix engineer who caused the data loss.

    Summary: "He did it." "No, he did it." "No, it was him!" "You did it FIRST!" "Idiot!" "Moron!" "Jackass!" ** customers shoot them both **

    --
    $nice = $webHosting + $domainNames + $sslCerts
  13. Re:Well DUH! by Chelloveck · · Score: 4, Funny

    I back up everything to /dev/null. Restoration is somebody else's job.

    --
    Chelloveck
    I give up on debugging. From now on, SIGSEGV is a feature.
  14. This isn't the first time & it won't be the la by mdonley · · Score: 2, Informative

    Does anybody remember Visto.com, formerly Briefcase.com? They had an incredible cloud storage and synchronization engine that I still have not found a suitable replacement for, and this was back in like 1998, way before Gmail and all the other glorious Google tools. They were so far ahead of their time it's ridiculous! Only 25mb of free storage, but still, back then, that was unheard of! Back in the day, their *free* service offered a tool you could download to your local machine, This tool would establish a local repository folder for files, which the tool would then handle the synchronization to the cloud. Not to mention integrating with outlook to synchronize address book, private calendars, public calendars, group calendars, tasks, bookmarks, and just about everything. Running this on two or more machines ensured that you were always in sync. Then, to top it off, they offered webmail, and website access to all of your files and bookmarks and everything else. Did I mention that it was *free*? But then the fateful day arrived when I was notified they were changing their business model to sell mobile phone synchronization software. I guess free services don't pay the bills. This was a company who was able to actually sue Microsoft for patent infringement and win... By the way, did anybody else out there in /. world use and like Visto? Has anybody ever found a suitable replacement? Google has a ton of different tools to help fill different areas of the void Visto.com left behind, but I don't think I will ever see another product like Visto come around again.

    --
    God look at me, I'm just a man, but you tell me I'm not just a man, so hard to understand, after all, I'm just a man.
  15. Storage Service Provider by HockeyPuck · · Score: 2, Informative

    I used to work at a Storage Service Provider back in the dotcom era (StorageNetworks baby!), and before you sign up for any type of service provider that would be providing access to your data you need to go over that contract with a fine toothed comb. Two areas that need to be covered are:

    What happens if the SSP goes out of business, how do you get your data back? In our contracts we would give you your data back either via access to the old arrays for X days to copy it somewhere else. Worst case: We drop a truckload of tapes off on your doorstep.

    Financial impact of loss of service. We had many financial customers that wanted to make sure we had "enough cash in the bank" to cover their financial losses if they suffered any downtime due to an issue on our end.

    Just like picking a hosting provider, you need to make sure you have contingency plans for data loss or corruption. If the SSP can't provide you with the services you need (backups, snapshots etc..), find another provider.

  16. Test restore, test restore, test restore! by gilgongo · · Score: 4, Insightful

    I don't have any way of verifying this story, but I worked with an old guy once who told me that he had been at a startup in the UK that was, by the sound of it, creating a kind of IMDB in about 1994. They had a team of researchers and a bunch of seed capital to create a large film database. Everything was ticking along for about 18 months and they had researched thousands of films.

    Then one day, the database shut down and they traced it to some bad hardware. They replaced the hardware and restored the database from the previous night's backup. Nothing doing - the backup tape (he said it was DAT) was corrupt. So they tried the other one. Nada. Same corruption. So they tried the off-site one. Same thing. Turned out all the backups they had made seem to have transferred the same corruption resulting in nothing significant recoverable.

    Had they tried a test restore at some point, they might have found out. As it was, a week after the crash, they shut the business down.

    Which reminds me of another (maybe apocryphal) story: the head of IT as a large company was fond of organising disaster recovery practices by walking into the data centre, physically removing a (pre-ordained) server and leaving a note in its place with the words "The server crashed" written on it. The support staff (and presumably management) knew that this would happen, but not when, or which machine (or dependent services) would be affected. Interesting test I would say.

    --
    "And the meaning of words; when they cease to function; when will it start worrying you?"