Slashdot Mirror
'Why You Should Not Use Google Cloud' (medium.com)
A user on Medium named "Punch a Server" says you should not use Google Cloud due to the "'no-warnings-given, abrupt way' they pull the plug on your entire system if they (or the machines) believe something is wrong." The user has a project running in production on Google Cloud (GCP) that is used to monitor hundreds of wind turbines and scores of solar plants scattered across 8 countries. When their project goes down, money is lost. An anonymous Slashdot reader shares the report: Early today morning (June 28, 2018) I receive an alert from Uptime Robot telling me my entire site is down. I receive a barrage of emails from Google saying there is some "potential suspicious activity" and all my systems have been turned off. EVERYTHING IS OFF. THE MACHINE HAS PULLED THE PLUG WITH NO WARNING. The site is down, app engine, databases are unreachable, multiple Firebases say I've been downgraded and therefore exceeded limits.
Customer service chat is off. There's no phone to call. I have an email asking me to fill in a form and upload a picture of the credit card and a government issued photo id of the card holder. Great, let's wake up the CFO who happens to be the card holder. What if the card holder is on leave and is unreachable for three days? We would have lost everything -- years of work -- millions of dollars in lost revenue. I fill in the form with the details and thankfully within 20 minutes all the services started coming alive. The first time this happened, we were down for a few hours. In all we lost everything for about an hour. An automated email arrives apologizing for "inconvenience" caused. Unfortunately The Machine has no understanding of the "quantum of inconvenience" caused.
Customer service chat is off. There's no phone to call. I have an email asking me to fill in a form and upload a picture of the credit card and a government issued photo id of the card holder. Great, let's wake up the CFO who happens to be the card holder. What if the card holder is on leave and is unreachable for three days? We would have lost everything -- years of work -- millions of dollars in lost revenue. I fill in the form with the details and thankfully within 20 minutes all the services started coming alive. The first time this happened, we were down for a few hours. In all we lost everything for about an hour. An automated email arrives apologizing for "inconvenience" caused. Unfortunately The Machine has no understanding of the "quantum of inconvenience" caused.
As if servers doing down can't happen if you host it yourself. Also, this isn't a problem limited to Google. We've been getting a lot of emails from Amazon lately with the subject " [Retirement Notification] Amazon EC2 Instance scheduled for retirement."
Our company tried to use Amazon a few years ago and ran into the same issues. Although google and amazon allow you to
spin up a single instance, they are really designed for companies that have hundred if not thousands of servers. Amazon
assumes that you have dozens of fault tolerant servers and if one goes down you just replace it with another one. This works
great for companies like Netflix but Amazon is a disaster for a company that isn't fully fault tolerant and has critical servers
that can't go down. Liquidweb, Rackspace, Linode, and even Digitalocean are more reliable when it comes to wanting to
keep a single server up and running with minimal downtime. Now if you need to keep thousands of servers up and don't care
if any one server goes down then Amazon works fine.
YouTube users, GMail users, etc. have all complained about similar issues with blackbox, zero accountability. On click, boom, you're done.
IANAL, but this is my theory...
We know that Google is controlled by some highly political people. People who want to be able to disconnect you, deplatform you, etc. at the drop of a dime. The more they make their services a customer service blackbox, the easier it is to get away with acting in bad faith.
By bad faith I mean specifically in contractual bad faith. All of the XKCD-citing hipsters miss a very important nuance of the law regarding "deplatforming assholes:" contracts are judged by the "good faith" conduct of both parties and evaluated by reasonable behavior standards.
They do things like tie your account to all of the services, including purchases, and after a few vague "bad behavior incidents" nuke it. Often taking real assets with them because of how those accounts are tied. I don't think, for instance, Microsoft would fair well if they cost someone $2k of XBox Live marketplace purchases because they cussed out a few butthurt players a few times (Microsoft claims it has the authority to do this). Google is the same way on a larger scale.
The more people that are involved, the more people who can be hauled into court, forced to testify, etc. You can demand they answer why they thought a reasonable person would act that way. You can point to flesh and blood people who are the focal point for a real user suffering real economic harm due to one or a few people's biases.
And then win damages.
IMO that is why you see these companies aggressively moving in this direction. It's about not facing as much accountability for acting like dicks.
Ah, yes. And that is at the core of the problem. But since this apparently happened before and they did not learn from that, my guess would be they have far larger problems than the known unreliability of cheap cloud services.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
This.
I'm retired now, but my firm had a plan to replace me with the cloud.
We were a law firm.
During my last two weeks, an elderly couple drove in from about 70 miles away to sign some family law papers and they were waiting in the conference room when a partner got hold of me and told me, "The cloud's down again .
I called the support number and they said they were aware of the problem and that they were working on it.
After a lot of pressure, I called again and told them to fail-safe over to the mirror that they had bragged about.
They said the outage got the mirror, as well.
I informed the partner and she started screaming at me. She yelled, "WHAT IS PLAN B?"
I said, "Ma'am, plan B is plan A."
It was quite a shit storm.
I had argued against the cloud, and I documented their rejection of my recommendations and they signed off on it.
They spent a a butt load of money bringing all that shit home, and I worked there another 5 years.
I had full hands-on control of the shit I built. All of it. I was totally responsible and could either fix, or get fixed, anything running in my house.
It little behooves the best of us to comment on the rest of us.
I have (neigh, had) a client making the same choice right now. All of their infrastructure, under my hand, I could fix as it was fully under my control. We'll see where they are in 5 years.
I'll happily go back to them when they ask me to clean up the shitstorm their current CTO doesn't see coming. As CTO and for double what I was getting as a contractor, of course.
Under my lead, they went from everything running on a single unstable server and going down every other day to everything running on a distributed cluster of servers and not a single outage in sight. This, in the matter of under a year, on a site that is their entire business and sees over 500k uniques and serves over 20 million pages per month.
Their new solution costs them more per month and limits them to 1.5 million pages served per month on the best package they actually list a price for, with no uptime SLA. In short, they'll be paying 15x as much for infrastructure for the capacity the current system has, which more than covers what they're "saving" by not paying me.
It is what it is, but they thought I was only looking out for my own ass when I pointed all of this out. Well, I was looking out for me but, as they were providing me steady full-time work, a huge part of that was looking out for them in-kind.
APK quotes people (including myself) without context and should not be trusted. Just thought you should know.
Yeah, he is right about that "single point of failure".
There are a huge number of companies that fall into that hole between "too small to have our own IT department" and "big enough to have a large IT department with formal procedures and personnel redundancies".
I was that "one guy who does everything" for a long time. It is easily the most efficient way to run a company. Even when we were a small department with a half-dozen really sharp guys, it was still a high-wire act, dependent on having A players who knew the business and systems well.
It was only after we hit maybe 15 or so IT employees that we really had full redundancy in personnel and reasonably well documented procedures. And even that was pretty small... we didn't have dedicated QA employees, for example.
It all depends on what your needs are and what your risk tolerance is. For most growing small companies, the reward of having the "one really smart guy" as a key point of failure is worth the risk. Later, when it comes time to sell out to an investor or go public, that risk/reward equation flips and you have to figure out how to eliminate that red flag on the audit.
As many here have said, going "cloud" isn't a panacea to fix that problem.
I signed a previous client up for Google Cloud services. I did so using the company credit card. I was in touch with Google pre-sales people throughout, and spoke to some account management as well. In other words, I did it correctly, with their oversight throughout.
What I'm saying is... you *can* pay for 'enterprise' services from Google with a credit card. Why anyone would want to is another matter, but to get invoice billing does take a few additional steps, which my client actually failed because of a change of business address at about the same time as we were doing the application. It all got resolved, as my client was in generally good standing, but I guess if your company is a bit flaky you might not pass such checks.
By the looks of things though, this particular company was using a consumer account, which is of course the source of the problem. If it also happened more than once, you'd think they'd have learned by now.
You get what you pay for. They thought they could get much more than they paid for. 100% their fault.
For my education, what would you point out as the flaws in the following analogy:
While it is "foreseeable" that I might have had different windows on a house I own, or that I'd be able to get out of the fire regardless of having the same windows because there are ground-floor doors in a house, is it really useful to talk about it being "100% my fault"? Should I, as a condo buyer, have had some kind of vague dread that makes me fundamentally at fault for whatever differences there are in the situation, and prevents me from talking about ways the building manager could improve things?
- First they ignore you, then they laugh at you, then ???, then profit.
"One of the strong points of cloud computing is the infrastructure to shift load to accommodate failing hardware. To reproduce that capability with your own hardware & infrastructure requires a tremendous amount of planning and capital investment: in power, servers, and network. It's almost never a simple matter of "fix the server immediately"."
Of course you need to pay for the required infrastructure. That is what "host yourself" means. Either you are paying the higher up front cost of building it out for yourself or you are paying the much higher costs to have google/amazon/digitalocean/etc do it for you. As for shifting load to accomodate failing hardware, it doesn't work as well as pitched (which is why these services go down all the time and have such high latency) but it works better and faster on a private cloud implementation. Which is where you've built a cloud stack on your own gear. Just be sure you actually have redundant controllers across the whole thing and didn't just build redundant controllers on a virtualized layer sitting on top of a not redundant system.