Slashdot Mirror
'Why You Should Not Use Google Cloud' (medium.com)
A user on Medium named "Punch a Server" says you should not use Google Cloud due to the "'no-warnings-given, abrupt way' they pull the plug on your entire system if they (or the machines) believe something is wrong." The user has a project running in production on Google Cloud (GCP) that is used to monitor hundreds of wind turbines and scores of solar plants scattered across 8 countries. When their project goes down, money is lost. An anonymous Slashdot reader shares the report: Early today morning (June 28, 2018) I receive an alert from Uptime Robot telling me my entire site is down. I receive a barrage of emails from Google saying there is some "potential suspicious activity" and all my systems have been turned off. EVERYTHING IS OFF. THE MACHINE HAS PULLED THE PLUG WITH NO WARNING. The site is down, app engine, databases are unreachable, multiple Firebases say I've been downgraded and therefore exceeded limits.
Customer service chat is off. There's no phone to call. I have an email asking me to fill in a form and upload a picture of the credit card and a government issued photo id of the card holder. Great, let's wake up the CFO who happens to be the card holder. What if the card holder is on leave and is unreachable for three days? We would have lost everything -- years of work -- millions of dollars in lost revenue. I fill in the form with the details and thankfully within 20 minutes all the services started coming alive. The first time this happened, we were down for a few hours. In all we lost everything for about an hour. An automated email arrives apologizing for "inconvenience" caused. Unfortunately The Machine has no understanding of the "quantum of inconvenience" caused.
Customer service chat is off. There's no phone to call. I have an email asking me to fill in a form and upload a picture of the credit card and a government issued photo id of the card holder. Great, let's wake up the CFO who happens to be the card holder. What if the card holder is on leave and is unreachable for three days? We would have lost everything -- years of work -- millions of dollars in lost revenue. I fill in the form with the details and thankfully within 20 minutes all the services started coming alive. The first time this happened, we were down for a few hours. In all we lost everything for about an hour. An automated email arrives apologizing for "inconvenience" caused. Unfortunately The Machine has no understanding of the "quantum of inconvenience" caused.
If millions of dollars are on the line, you should be running your own systems. Seriously. I'm not an IT expert, data infrastructure guy or anything. I'm just a dumb nerd, and I know that. Never trust your data to a third party when millions are at stake -- let alone critical infrastructure reliability.
Beware of the Leopard.
Why was there a second time?
Our company tried to use Amazon a few years ago and ran into the same issues. Although google and amazon allow you to
spin up a single instance, they are really designed for companies that have hundred if not thousands of servers. Amazon
assumes that you have dozens of fault tolerant servers and if one goes down you just replace it with another one. This works
great for companies like Netflix but Amazon is a disaster for a company that isn't fully fault tolerant and has critical servers
that can't go down. Liquidweb, Rackspace, Linode, and even Digitalocean are more reliable when it comes to wanting to
keep a single server up and running with minimal downtime. Now if you need to keep thousands of servers up and don't care
if any one server goes down then Amazon works fine.
If an extended system outage can cause "millions of dollars in lost revenue" then you should have a DR plan. Don't put all your eggs in one basket. Have copies of everything at another site (EC2, Azure, Colo, etc) that you can turn on and switch to in this event. If millions of dollars are on the line, then it shouldn't be unreasonable to have such a plan and infrastructure established.
I have an email asking me to fill in a form and upload a picture of the credit card and a government issued photo id of the card holder. Great, let's wake up the CFO who happens to be the card holder. What if the card holder is on leave and is unreachable for three days?
Uh, I don't know - take a picture of each and save them on your phone, in case you need them?
You report everything was back up within 20 minutes once you submitted the requested information - that seems pretty good to me.
Now, about your decision to only run one instance of your mission critical application suite on exactly one cloud service...
The story here is you consider it someone else's fault for your failure to plan/prepare for an outage.
Ken
I have an email asking me to fill in a form and upload a picture of the credit card and a government issued photo id of the card holder. Great, let's wake up the CFO who happens to be the card holder. What if the card holder is on leave and is unreachable for three days? We would have lost everything -- years of work -- millions of dollars in lost revenue.
Somewhere in Russia, India and Nigeria, several callcenters full of scammers came all at once.
-=This sig has nothing to do with my comment. Move along now=-
Why was there a second time?
So many of the problems here (ex. paying with a credit card and one that has only a single person's name on it? Having no fallback that can be spun up elsewhere?) are foolish if this has never happened before, and utterly, mind-bogglingly idiotic if this in fact has already happened before. It's one thing to be blind of something you should know could be a problem, it's quite another to be blind and wholly unprepared for a problem you've personally experienced! Something seems fundamentally wrong at this company.
Also, if your entire business can die because it takes an unexpected few days off, then perhaps your business is running a bit too raggedly and doesn't have enough meat on the bones . . .
I remember sigs. Oh, a simpler time!
The company thought they could get away with paying less for server infrastructure. They can. But they get less. This is one of the "less" things they get.
If you value your data, host it yourself, preferably in multiple locations. If you want to go cheap, then you can expect to lose things.
Like your data, or access to it, or availability of it.
It's not such a smart thing to cheap out on the important stuff.
Of course, convincing the bean counters of future risk inherent in what appears to them to be current savings... good luck with that.
Well, best to get rid of your bean counters. :)
Here's a maxim of mine I like to drop on the table during discussions like these:
I've fallen off your lawn, and I can't get up.
I may be one of the "old timers" who I'm told is thinking about things in an "old school" way when I say this. But I've *always* warned people that "The Cloud" just means you're giving somebody else the responsibility of handling your data and the systems it runs on.
That makes sense sometimes. I'm not "anti cloud". But for anything really critically important to a business, I feel you should have it running locally and THEN consider cloud options as hot-failover sites, backup sites, etc. With cloud hosting, the whole thing is off limits to you as soon as your Internet circuit goes down, for one thing. With it running locally, you can still use it just fine anywhere on your LAN.
But additionally, if the provider hosting your stuff goes bankrupt or merges with someone else, or just plain decides it's not profitable enough without some pricing changes -- where does that leave you? Technically, they can just disappear with your whole software and data configuration overnight. Or they can put trained apes in charge of maintaining things so it suddenly has huge security holes. Who knows?
When you run things yourself, YOU are where the buck stops if things go wrong. If you're good at what you do, that should be more of a comforting thing than a scary thing. I've seen too many shops trying to cut corners on the I.T. hiring budget by bringing in less experienced people who really can't properly run the systems they're supposed to be caring for. The cloud for them is a crutch ... a way to get things done that are beyond their abilities. But that's not an ideal situation for a business to put itself in.
This sounds like asking fro trouble to me!
This is a good time to remind people that XKCD wrote that comic to justify the forceful expulsion from the entire Internet of ordinary people
No it's a good time to remind people you've gone way off the deep end, mate.
It does however prove that just aobut anything in favour of gamergate no matter how batshit insane will get modded up here. Like this bit:
of al-Qaeda.
Aside: isn't that sort of old news even for the crazies? Aren't ISIS responsible for chemtrails now or are they merely a false flag perpertrated by the deep state t ostop us knowing the truth about how a chemtrail spraying plane actually did 9/11?
All of that "Gamergate harassment?" There was one tweet.
Gamergate was one tweet: +3 Insightful. I think that might be a new low for slashdot moderation.
SJW n. One who posts facts.