Slashdot Mirror

← Back to Stories (view on slashdot.org)

'Why You Should Not Use Google Cloud' (medium.com)

Posted by BeauHD on from the time-is-money dept.

A user on Medium named "Punch a Server" says you should not use Google Cloud due to the "'no-warnings-given, abrupt way' they pull the plug on your entire system if they (or the machines) believe something is wrong." The user has a project running in production on Google Cloud (GCP) that is used to monitor hundreds of wind turbines and scores of solar plants scattered across 8 countries. When their project goes down, money is lost. An anonymous Slashdot reader shares the report: Early today morning (June 28, 2018) I receive an alert from Uptime Robot telling me my entire site is down. I receive a barrage of emails from Google saying there is some "potential suspicious activity" and all my systems have been turned off. EVERYTHING IS OFF. THE MACHINE HAS PULLED THE PLUG WITH NO WARNING. The site is down, app engine, databases are unreachable, multiple Firebases say I've been downgraded and therefore exceeded limits.

Customer service chat is off. There's no phone to call. I have an email asking me to fill in a form and upload a picture of the credit card and a government issued photo id of the card holder. Great, let's wake up the CFO who happens to be the card holder. What if the card holder is on leave and is unreachable for three days? We would have lost everything -- years of work -- millions of dollars in lost revenue. I fill in the form with the details and thankfully within 20 minutes all the services started coming alive. The first time this happened, we were down for a few hours. In all we lost everything for about an hour. An automated email arrives apologizing for "inconvenience" caused. Unfortunately The Machine has no understanding of the "quantum of inconvenience" caused.

18 of 508 comments (clear)

  1. Sorry, but... by Known+Nutter · · Score: 5, Insightful

    If millions of dollars are on the line, you should be running your own systems. Seriously. I'm not an IT expert, data infrastructure guy or anything. I'm just a dumb nerd, and I know that. Never trust your data to a third party when millions are at stake -- let alone critical infrastructure reliability.

    --
    Beware of the Leopard.
    1. Re:Sorry, but... by snapsnap · · Score: 5, Interesting

      As if servers doing down can't happen if you host it yourself. Also, this isn't a problem limited to Google. We've been getting a lot of emails from Amazon lately with the subject " [Retirement Notification] Amazon EC2 Instance scheduled for retirement."

    2. Re:Sorry, but... by Nutria · · Score: 5, Insightful

      As if servers doing down can't happen if you host it yourself.

      But then you're in control, instead of having to rely on some amorphous, anonymous monster that only allows communication via automated email.

      --
      "I don't know, therefore Aliens" Wafflebox1
    3. Re:Sorry, but... by Anonymous Coward · · Score: 5, Insightful

      That's why you also configure a failover at a secondary location. This server punching guy sounds like a damn idiot. With millions on the line why is there no disaster recovery plan outside of "call the CFO for his credit card?"

    4. Re:Sorry, but... by AmiMoJo · · Score: 5, Insightful

      Google's cloud services are enterprise grade if you pay enterprise prices for them.

      If you pay for it on a credit card assigned to the CFO then you are not an enterprise and you are not paying enterprise prices.

      They chose a cheap, no-SLA no-support service, probably because it was cheap. Then they get upset that they aren't receiving the support they didn't pay for.

      --
      const int one = 65536; (Silvermoon, Texture.cs)
      SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
    5. Re:Sorry, but... by Anonymous Coward · · Score: 5, Insightful

      As if servers doing down can't happen if you host it yourself.

      But then you're in control, instead of having to rely on some amorphous, anonymous monster that only allows communication via automated email.

      Exactly right. If your servers go down, YOU can fix them immediately and you don't have the problem of trying to get in touch with some support person who may or may not respond quickly, or may not respond at all if you can't even figure out how to get in touch with them.

      That said, this was a case of a company trying to be cheap, and instead of an enterprise account they tried to run critical applications on what is supposed to be a consumer-level account, which is why they started getting e-mails from Google about "suspicious activity".

    6. Re:Sorry, but... by Anonymous Coward · · Score: 5, Informative

      I can understand summarily shutting down a customer on a residential Internet connection, or a small business shared web hosting provider. However when providing an "enterprise grade" service, you should be prepared to give your customers the benefit of the doubt. About the only instances I can think of for an enterprise service to shut down a customer is if they are greatly exceeding their allocated resources and/or the activity associated with the customer is actively in the process of harming other customers.

      Cheapfuck customer was running enterprise-level applications on what was supposed to be a consumer-level account. That's why Google Bots detected "suspicious" actitivy and shut them down. If they hadn't tried to be cheapfucks and were using an enterprise account this most likely wouldn't have happened.

    7. Re:Sorry, but... by gweihir · · Score: 5, Interesting

      Ah, yes. And that is at the core of the problem. But since this apparently happened before and they did not learn from that, my guess would be they have far larger problems than the known unreliability of cheap cloud services.

      --
      Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
    8. Re:Sorry, but... by Peter+P+Peters · · Score: 5, Informative

      Google's cloud services are enterprise grade if you pay enterprise prices for them.

      If you pay for it on a credit card assigned to the CFO then you are not an enterprise and you are not paying enterprise prices.

      They chose a cheap, no-SLA no-support service, probably because it was cheap. Then they get upset that they aren't receiving the support they didn't pay for.

      Exactly this. I don't use Google Cloud, but my 'enterprise' AWS service comes with a dedicated account manager and architect who I can call at anytime for help.
      Sounds like this guy cheaped out then is bitching because he received cheap service.

  2. Wait by Anonymous Coward · · Score: 5, Insightful

    The first time this happened,...

    Why was there a second time?

  3. Amazon's cloud s no better by Wycliffe · · Score: 5, Interesting

    Our company tried to use Amazon a few years ago and ran into the same issues. Although google and amazon allow you to
    spin up a single instance, they are really designed for companies that have hundred if not thousands of servers. Amazon
    assumes that you have dozens of fault tolerant servers and if one goes down you just replace it with another one. This works
    great for companies like Netflix but Amazon is a disaster for a company that isn't fully fault tolerant and has critical servers
    that can't go down. Liquidweb, Rackspace, Linode, and even Digitalocean are more reliable when it comes to wanting to
    keep a single server up and running with minimal downtime. Now if you need to keep thousands of servers up and don't care
    if any one server goes down then Amazon works fine.

    1. Re:Amazon's cloud s no better by fahrbot-bot · · Score: 5, Insightful

      ... but Amazon is a disaster for a company that isn't fully fault tolerant and has critical servers that can't go down.

      If your company has "critical servers that can't go down" (wherever they are) and you're not fully fault-tolerant, you're the disaster, not Amazon.

      Not here to pick a fight, just sayin'. (one finger pointing at someone else is also three fingers pointing at yourself)

      --
      It must have been something you assimilated. . . .
  4. Yeah, big warning signs on the user side here by Phil+Urich · · Score: 5, Insightful

    The first time this happened,...

    Why was there a second time?

    So many of the problems here (ex. paying with a credit card and one that has only a single person's name on it? Having no fallback that can be spun up elsewhere?) are foolish if this has never happened before, and utterly, mind-bogglingly idiotic if this in fact has already happened before. It's one thing to be blind of something you should know could be a problem, it's quite another to be blind and wholly unprepared for a problem you've personally experienced! Something seems fundamentally wrong at this company.

    Also, if your entire business can die because it takes an unexpected few days off, then perhaps your business is running a bit too raggedly and doesn't have enough meat on the bones . . .

    --
    I remember sigs. Oh, a simpler time!
  5. Cheap service, cheap results by fyngyrz · · Score: 5, Insightful

    The company thought they could get away with paying less for server infrastructure. They can. But they get less. This is one of the "less" things they get.

    If you value your data, host it yourself, preferably in multiple locations. If you want to go cheap, then you can expect to lose things.

    Like your data, or access to it, or availability of it.

    It's not such a smart thing to cheap out on the important stuff.

    Of course, convincing the bean counters of future risk inherent in what appears to them to be current savings... good luck with that.

    Well, best to get rid of your bean counters. :)

    Here's a maxim of mine I like to drop on the table during discussions like these:

    If you can't afford to do it well, you almost certainly shouldn't be doing it at all.

    --
    I've fallen off your lawn, and I can't get up.
    1. Re:Cheap service, cheap results by CaptainDork · · Score: 5, Interesting

      This.

      I'm retired now, but my firm had a plan to replace me with the cloud.

      We were a law firm.

      During my last two weeks, an elderly couple drove in from about 70 miles away to sign some family law papers and they were waiting in the conference room when a partner got hold of me and told me, "The cloud's down again .

      I called the support number and they said they were aware of the problem and that they were working on it.

      After a lot of pressure, I called again and told them to fail-safe over to the mirror that they had bragged about.

      They said the outage got the mirror, as well.

      I informed the partner and she started screaming at me. She yelled, "WHAT IS PLAN B?"

      I said, "Ma'am, plan B is plan A."

      It was quite a shit storm.

      I had argued against the cloud, and I documented their rejection of my recommendations and they signed off on it.

      They spent a a butt load of money bringing all that shit home, and I worked there another 5 years.

      I had full hands-on control of the shit I built. All of it. I was totally responsible and could either fix, or get fixed, anything running in my house.

      --
      It little behooves the best of us to comment on the rest of us.
    2. Re:Cheap service, cheap results by BronsCon · · Score: 5, Interesting

      I have (neigh, had) a client making the same choice right now. All of their infrastructure, under my hand, I could fix as it was fully under my control. We'll see where they are in 5 years.

      I'll happily go back to them when they ask me to clean up the shitstorm their current CTO doesn't see coming. As CTO and for double what I was getting as a contractor, of course.

      Under my lead, they went from everything running on a single unstable server and going down every other day to everything running on a distributed cluster of servers and not a single outage in sight. This, in the matter of under a year, on a site that is their entire business and sees over 500k uniques and serves over 20 million pages per month.

      Their new solution costs them more per month and limits them to 1.5 million pages served per month on the best package they actually list a price for, with no uptime SLA. In short, they'll be paying 15x as much for infrastructure for the capacity the current system has, which more than covers what they're "saving" by not paying me.

      It is what it is, but they thought I was only looking out for my own ass when I pointed all of this out. Well, I was looking out for me but, as they were providing me steady full-time work, a huge part of that was looking out for them in-kind.

      --
      APK quotes people (including myself) without context and should not be trusted. Just thought you should know.
    3. Re:Cheap service, cheap results by aquabat · · Score: 5, Funny

      I have (neigh, had) a client making the same choice right now.

      +1 for having the horse sense to move on from that client, after you told them to hold their horses and they didn't listen. I guess you can lead a horse to water, but you can't make him drink. If they're going to get on their high horse and be as stubborn as a mule, then they're going to end up trying to close the barn door after the horse has bolted. When comparing in-house vs cloud, in-house really is a horse of a different colour. Trying to run a tech business on the cloud is just horsing around.

      --
      A republic cannot succeed till it contains a certain body of men imbued with the principles of justice and honour.
  6. If you look at the first comment in TFA by Chrisq · · Score: 5, Informative
    If you look at the first comment in TFA it turns out that the poster was using a consumer account for a million-dollar mission critical application:

    I highly recommend establishing an enterprise relationship with Google Cloud. It seems you are running a mission critical application on a consumer account and this issue could have been avoided. Reach out to the support team and let them know you want to discuss enterprise options to ensure you have done everything possible to ensure your account is never impacted like this in the future. Ping me if you have any trouble getting through. - Mike Kahn Customer Engineer, Google Cloud. All views and opinions are my own. @mkahn5

    This sounds like asking fro trouble to me!