Slashdot Mirror
Why Auto-Scaling In the Cloud Is a Bad Idea
George Reese writes "It seems a lot of people are mistaking the very valuable benefit that cloud computing enables — dynamically scaling your infrastructure — with the potentially dangerous ability to scale your infrastructure automatically in real-time based on actual demand. An O'Reilly blog entry discusses why auto-scaling is not as cool a feature as you might think."
While a content site might run the risk of getting slashdotted or Dugg, that isn't necessarily a big risk for applications. And your platform choice makes a big difference. We do our business applications on a LAMP stack. If we need capacity, we can stand it up for the cost of hardware. Nice thing about LAMP is at least the AMP part is OS portable, so we can rent capacity where ever it's cheap. So far we haven't needed to do that but it's nice to have the ability.
To date we haven't run into any problems. If we're expecting a surge of new customers, we have a pretty good idea of expected traffic per customer. We can stand up the capacity well in advance. Hardware is cheap and can be repurposed if end up not needing all the extra capacity.
Our platform choice gives us a tremendous amount of flexibility. You don't get that with Windows. Any increase in capacity has a significant price tag in license fees associated with it. Once you build the capacity there are fairly significant ongoing expenses to maintain it. You can take it offline if you need to scale down but you don't get your money back on the licenses. There's a whole new set of problems outsourcing your hosting.
I like our setup. The flexibility, the scalability, the peace of mind of not struggling with capacity issues, negotiating license agreements with MS or one of their solution providers and not being limited to their development environment. We can build out a lot of excess capacity and just leave it sit in the rack. If we need more just push a button and light it up. I'm not sure an Amazon or anyone else could do it cheap enough to justify moving it. And I really like having the extra cash. Cash is good. Peace of mind and extra money...what's not to like? Keep your cloud.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
His complaint with auto-scaling was that if the org is doing their proverbial homework then and planning for additional capacity then they should not need it.
There are times when traffic boosts come as a bit of a surprise. However, depending on size and free capacity some bumps should be able to smooth out.
Another trick is to have the means to scale some functionality down to allow for additional traffic. Slashdot for instance used to flip to a static front page when traffic was insane.
Personally, a very limited automatic scale to meet a few percentage points might not be a bad idea at least to create additional buffer for increased reaction time. Still, alarms should sound and I would think of this as a fall back option.
All in all, I rather agree with his sentiment. Don't be lazy and don't waste cash.
"You should always go to other people's funerals; otherwise, they won't come to yours." -- Yogi Berra
What if someone posts a bad batch or accidently malforms some package in such a way to chew though 10x the resources.
I think there are many great uses for cloud environments, but people have to be careful when it is pay for play.
It's a bit different then tying up all the resources on the web server. Sure, there is cost in time, but rarely does anyone get billed for those man hours.
"You should always go to other people's funerals; otherwise, they won't come to yours." -- Yogi Berra
First Rule of Automation: Automation applied to an efficient task increases it's efficiency; likewise, automation applied to an inefficient task will simply increase the problem until it's an all out clusterfuck.
Last time I checked, most sites that get slashdotted are either some shiatty shared hosting or a dynamic page.
Static pages & CoralCDN would keep a lot of websites from getting hammered off the internet.
[Fuck Beta]
o0t!
i think the author's point is that dynamic scaling should always be planned; partly because it results in better understanding of traffic patterns, and thus better long-term capacity planning, and partly because you need to be able to distinguish between valid traffic and DDoS attacks. still, i think the author is overstating it a bit. one of the main draws of cloud computing to smaller businesses is the ability to pool resources more efficiently through multitenancy, part of which is precisely due to auto-scaling. without the cloud being able to dynamically allocate resources to different applications as needed in real-time (i.e. without human intervention), there isn't much of an advantage to sharing a cloud infrastructure over leasing dedicated servers.
for instance, let's say there are 10 different startups with similar hosting needs, and they can each afford to lease 10 application servers on their own. so using traditional hosting models they would each lease 10 servers and balance the load between the them. but after a few months they realize that 75% of the time they only really need 5 servers, and 20% of the time they need all 10, but an occasional 5% of the time they need more than 10 servers to adequately handle their user traffic. this means that in their current arrangement, they're wasting money on more computing resources than they actually need most of the time, and yet they still have service availability issues during peak loads 5% of the time (that's over 2.5 weeks a year).
all 10 of these startups share a common problem--they each have variable/fluctuating traffic loads severely reducing server utilization & efficiency. luckily, cloud computing allows them to pool their resources together. since the majority of the time each startup needs only 5 servers, the minimum number of virtual servers their cloud infrastructure needs is 50. and since each startup needs double that 20% of the time, 10 extra virtual servers are needed (shared through auto-scaling). but since each startup needs more than 10 servers for about 2.5 weeks each year, we'll add another 15 extra virtual servers. so all in total, the 10 startups are now sharing the equivalent of 75 servers in their cloud.
by hosting their applications together on a cloud network, each startup not only has their hosting needs better met, but they also stand to save a lot of money because of better server utilization. and each startup now has access to up to 30 virtual servers when their application requires it. this kind of efficiency would not be possible without a cloud infrastructure and auto-scaling.
On the surface auto-scaling is obviously a great thing. But it doesn't take much thought to start punching holes in it.
Lets first look at the Data center that provides such a glorious capability.
1. It is their own best interest for you to scale up. Scale up processing, disk, bandwidth or what ever. For the simple reason it's more money. Since you signed the contract you will probably be scaled well and truly before you know it. Usually you only find out when the bill comes in.
2. The data center has very little incentive to make sure you are notified in a timely manor of autoscaling. As a matter of fact this feature is usually crippled or even broken. I don't care what the contract says. The datacenter rarely honors this part of the contract to anyones satisfaction.
Now lets look at the client and the horrible things that can go wrong. By no means even remotely a complete list.
The new version of the app list.
1. Bob the developer forgets to index that new DB table. Database goes nuts trying to do a simple select. BAM autoscaling of DB CPU resources goes through the roof.
2. New AJAX call is not properly tested. For some reason it now triggers div refreshes as the mouse moves. App server is now flooded. BAM Band Width and CPU autoscale through the roof.
3. App no longer properly caches that all important query. BAM again DB and APP CPU skyrockets.
4. The genius in dev decides to make the jsession stateful. Works fine on the desktop. Works fine when load test hammers only 10 users. Oh Oh real world kicks in, in Prod. We have 10k users. Everything goes through the roof.
5. The list of new version issues goes on.
The bad guys come a knocking.
OK so now your a hot property on the net and you sign up for autoscaling so that you don't have to worry about capacity planning. You are focused on that cash machine that is your cool app.
1. You didn't know about that monster hole in the app. The bad guys inject a phishing site onto your Uber site. The phishing site is wildly successful. Oh crap we just paid for the biggest fraud site on the net.
2. The dev team leaves that back door on the site so they can maintain it remotely. Oh Oh all of a sudden you notice port 25 traffic is off the charts from the site. OMG we just uploaded 25Tbtyes in the last 24hours. You have just joined the ranks of the largest SPAM generators on the planet. You have a monster bandwidth bill and a very expensive legal bill.
3. What are these very large globs in the database all of a sudden. OH crap we left a hole and are vulnerable to SQL injection. OH crap it's all encrypted kiddie porn. Bills for bandwidth, disk and legal come a knocking.
I do have experience with this sort of thing. And it always goes sour at some point. The techies are always overruled by the marketing and business types on this. As the deal is always so great on paper. At some point something will go wrong. Software is never perfect. Between defects and bad guys you are a sitting duck for the big man carrying the bill to your door. It's only ever a matter of time.
Oh and lastly. Geuss what some times the autoscaling fails. Make that a lot of the time. And you are then off the air.
The best situation is for you as a customer of scaling is to have a close relationship with the supplier. Once you start to reach certain predefined levels of usage they should contact you and give you the option of an upgrade. Make the scaling feature by human choice. Never let the supplier decide that for you.
Well yes, you could also look at it from the point of view of. "I have a really clever idea, which will probably take off, and which if it does take off will require a lot of resources. I don't have a lot of money, but I can scrape together the cash for a small cloud investment and if my idea takes off I can afford as many servers as I want. I could buy a couple of regular servers and be unable to meet demand for several weeks while I order new equipment and possibly lose my start because people got sick of my site not being up, I could sell my idea to some venture capital people who, if they invest at all will take half my profits, or I can use the cloud, expand in ten minutes, and maybe make a lot of money without having to give it all to someone else".
That's the strength of the cloud my friend, being able to start an idea without having to promise 90% of it to someone else to get funding.
Yeah, it seems like is argument really comes down to a couple points:
So if someone offered a service where auto-scaling was fast, and there was some kind of limits on what you could be charged under what sorts of situations, would he still have a problem with auto-scaling? I was expecting something a little more absolute, like "there's a definite trade-off between security and accessibility", but it seems like he's saying something more like, "Right now there's no service that is offering auto-scaling services that are good enough."
I have. My company lives (or dies) by the !@# SLA.
Our agreements require no less than 99.9% uptime, about 8 hours of downtime per year. We never gotten close to that - our worst year was about 2.5 hours of downtime because of a power failure at our "fully redundant" hosting facility.
In this world, where I have up to 8 hours per year, 10 minute response would be a god-send. We've just spent *alot* of money revamping our primary cluster so that we now operate with 100% full redundancy on everything. Redundant network feeds. Redundant logic servers. Redundant load balancers. Redundant database servers. All with auto failure, dynamic routing with DNS. (which is, itself, very failure tolerant)
But an application has to be constructed in a very particular way in order to scale, particularly if data integrity is important. (EG: ACID compliance SQL) This is often counter-intuitive and non-obvious, and porting an existing application to such an environment is not a quick investment. It's very typical to give up raw performance for performance scalability. We've devoted approximately 6 man-months over the past year to take full advantage of clustered, redundant computing in order to try for 1 hour over the next year along with near-linear scalability.
It's not just about capacity - it's about keeping all those !@# servers organized and coordinated!
Bottom line? Take a look at your SLA.
In our case, if we suffered a few hours of downtime every year or so, it would be an inconvenience to our users and clients. In any event, our uptime is best-of-breed in our niche-ish industry, but I'd put our uptime as mid range for hosted products overall, when you include companies that are much bigger than our still-somewhat-small rapidly-maturing startup.
Spend money where it counts. This requires an understanding of your economic base. If somebody slashdots your site, is that your golden opportunity, or is that an annoyance. In our case, a few hours of downtime if we got slashdotted wouldn't cause any particular long-term problem if it brought us down. If you have a few hundred customers paying $10/month for some cheap-o websites, a few hours of downtime every year or two won't cause much problem.
I have no problem with your religion until you decide it's reason to deprive others of the truth.
This reminds me of a large company which outsourced enthusiastically, until at one point they discovered they'd outsourced decisions about maintenance... causing the outsourcer to have control over the maintenance budget.
As you might expect, after it ballooned, they started in-sourcing!
Giving others control over financial decisions is almost always unwise, even if doing so is the newest, coolest idea of the week.
--dave
davecb@spamcop.net
Of course, you can only do this if you know you're under attack, and if your infrastructure is set to autoscale, you probably won't know. Until you receive the bill.
Yes because if you happen to use some sort of auto-scaling system, be it at the cloud level or your own management system, it's very likely that you never thought to put in the same monitoring and alerting systems that you already had on your non-cloud, non-autoscaling systems thus ensuring that you will be blindsided by the scenario you just laid out.
Or, you have more than two brain cells to rub together and you already had all of that in place and just pointed it to the auto-scaling cloud system enabling you to react the same way, except without the downtime in the middle.
As a cloud computing provider, I actually have no interest in having my customers suddenly run up huge bills. The reason is that as the article said, something is most likely wrong somewhere, which means that as their services provider, I'll also be responsible for figuring it out :) I can't speak for Amazon, which has a more hands-off model, but my success is invested in the success of my customers, so I won't sit idly by while they waste their money.
However, looking at my company's balance sheet, we make our money off of base load, not peaks. Unlike what one of the other posters said, we can't average the peak load across customers since most customers have peaks at the same time, so accommodating peak load is more of a money-losing proposition since the bulk of those servers lie idle much of the day. In a truly round-the-clock (geographically distributed) cloud operation, this might be true, but even Amazon, which makes you choose the continent you want to run your cloud servers in, still has to hold a lot of reserve capacity (which is built into their rates) to accommodate the usual twice-daily peak loads.
For web sites, a peak load that is many times the base load usually indicates something is wrong with the business model as well as the software, since SaaS providers also can't make any money off short bursts of usage. In many cases, peaks that last less than the provisioning time of a new instance (which is typically no less than a few minutes because of the time to load the instance's memory from storage) have to be handled differently anyway, either with more base allocation or for example with a queue of work to be done and notifications to customers when that work is completed.
there would be various triggers of "if capacity exceeds A in time frame B, someone gets emailed/paged and is given the opportunity to override."
Point is, the overriding should probably happen after the system has attempted to auto-scale.
For instance, if I got Slashdotted, I'd probably want to scale to handle the load. If I have to be called in to make a decision before any scaling happens, I've probably missed an opportunity. On the other hand, if I've set reasonable limits, I then have the choice to relax some of those limits, or to decide I can't afford surviving Slashdot this time (or maybe realize it's a DDOS and not Slashdot), and pulling the plug -- but an hour's worth of extra capacity shouldn't kill me.
Of course, that all depends on what kind of site you're running. Some sites might rather be taken completely down by a Slashdotting than spend too much on hosting.
Don't thank God, thank a doctor!