Slashdot Mirror
Why Auto-Scaling In the Cloud Is a Bad Idea
George Reese writes "It seems a lot of people are mistaking the very valuable benefit that cloud computing enables — dynamically scaling your infrastructure — with the potentially dangerous ability to scale your infrastructure automatically in real-time based on actual demand. An O'Reilly blog entry discusses why auto-scaling is not as cool a feature as you might think."
I think auto-scaling the clouds based on actual demand is a really great idea. I think farmers would really like that feature, in fact.
Wait, what clouds?!
Without a hard-limit, some people run up big cell-phone bills. If you are forced to stop and plan and budget when you exceed resources, then you have better control over them. Cloud companies will likely not make metering very easy or cheap because they *want* you to get carried away.
Table-ized A.I.
THe author states that one reason he doesn't like autoscaling is because it can take a while to take effect. Thats bad technology, waiting for someone to come along and improve it.
He also says he doesnt like autoscaling even with limiters. Autoscaling with limiters makes sense to me, especially if the limits are things along the line of 'dont spend more than XXX over time Y'.
Finally, not using autoscaling because you might get DDoS'd is just stupid. You lose business/visitors. Thats worse than paying more to avoid being taken down, because your reputation gets hurt AS WELL AS losing you business.
The blogosphere has disagreed with the use of web2.0 in the cloud. Sure, we all know that data is king and that's why we use software as a service nowadays with the web as a platform using AJAX and RSS extensively. This has helped to solve the challenge of findability since lightweight companies helps to connect user needs. The fact is that the long tail is part of the paradigm of user as co-developers in server wiki-like sites. Unfortunately this brings up the problem of ownership of user generated content. But I think that perpetual betas help the architecture of participation to stimulate web2.0. Interaction does make the experience good.
Knowledge is power. Knowledge shared is power lost.
Someone get this guy a cane to shake at the whipper-snappers. "In my day, you learned proper capacity planning or you didn't enter the data center!"
It can take up to 10 minutes for your EC2 instances to launch. That's 10 minutes between when your cloud infrastructure management tool detects the need for extra capacity and the time when that capacity is actually available. That's 10 minutes of impaired performance for your customers (or perhaps even 10 minutes of downtime).
Like, you could do it so much faster than 10 minutes without auto-scaling. Bah! If you've read The Art of Capacity Planning you would've mailed in the coupon for the free crystal ball and seen this coming!
Properly used, automation is a good thing. Blindly relying on it will get you burned, but to totally dismiss it out of hand is foolish.
Learning HOW to think is more important than learning WHAT to think.
So I hand over my business logic and data to a third party, who may or may not meet a promised SLA, and whose security I cannot verify? Does this mean I can be rooted and lose my customer data faster, and at a rate proportional to the hack attempts? Cool!
I want to delete my account but Slashdot doesn't allow it.
While a content site might run the risk of getting slashdotted or Dugg, that isn't necessarily a big risk for applications. And your platform choice makes a big difference. We do our business applications on a LAMP stack. If we need capacity, we can stand it up for the cost of hardware. Nice thing about LAMP is at least the AMP part is OS portable, so we can rent capacity where ever it's cheap. So far we haven't needed to do that but it's nice to have the ability.
To date we haven't run into any problems. If we're expecting a surge of new customers, we have a pretty good idea of expected traffic per customer. We can stand up the capacity well in advance. Hardware is cheap and can be repurposed if end up not needing all the extra capacity.
Our platform choice gives us a tremendous amount of flexibility. You don't get that with Windows. Any increase in capacity has a significant price tag in license fees associated with it. Once you build the capacity there are fairly significant ongoing expenses to maintain it. You can take it offline if you need to scale down but you don't get your money back on the licenses. There's a whole new set of problems outsourcing your hosting.
I like our setup. The flexibility, the scalability, the peace of mind of not struggling with capacity issues, negotiating license agreements with MS or one of their solution providers and not being limited to their development environment. We can build out a lot of excess capacity and just leave it sit in the rack. If we need more just push a button and light it up. I'm not sure an Amazon or anyone else could do it cheap enough to justify moving it. And I really like having the extra cash. Cash is good. Peace of mind and extra money...what's not to like? Keep your cloud.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
He seems to be assuming that you only want to run a website on this service. I don't think hosting websites on this kind of service is a good idea at all. There are many other types of application you run on clould computing infrastructure, which makes much more sense, and negates almost all of his claims.
Consider for example a rendering farm. One day you may have two items to render. Another day 10 items. The next day 5 items. Should you really scale up and down manually each day, when you could just as easily just start the amount of servers you need based on how many jobs have been submitted for that day, and how large the jobs are?
There are many other examples. Websites are not the only thing you run on these services.
Posted by a Debian GNU/Linux user
Yeap, that's right. With over 7yrs of solid hosting industry experience, it's very easy to see.
Atleast Amazon's service is WAY overpriced for long term use. Sure if you need it just for few hours ever it's all good, but for 24/7 hosting it ain't, none of them.
It's cheaper to get regular servers, even from a very high quality provide than to use amazon's services.
Best of all: You can still use their service to autoscale up if you prepare right, and yet have low baseline cost.
If it's only filehosting service you need, the BW prices amazon offers are outrageous, take a bunch of cheapend shared accounts, and you'll get way better ROI, and still, for the most part, do not sacrifice any reliability at all. Cost: Greater setup time, depending upon on several contingency factors.
Case examples: you can get from bluehost, dreamhost etc. plenty of HDD & Bandwidth for few $ a month. Don't even try to run any regular website on it, they'll cut you off (CPU & Ram usage), but for filehosting, it's great bang for buck :)
Scared of reliability? Automatically edit DNS zone according to locations availability and have low(ish) TTL. Every added location increases reliability.
Pulsed Media Seedboxes
I did some rough cost comparisons for a high-traffic web site in my similarly cynical article a few weeks ago (disclaimer: I run a hosting company flogging unfashionable servers, and am not a cloud fan yet :) ).
Matthew @ Bytemark Hosting
His argument basically boils down to "Auto-scaling is a bad idea because you might implement it badly and then it will do the wrong thing". Isn't that true of everything? The flip side, is that if you implement it well, then auto-scaling would be a great idea!
It's like saying that dynamically sized logical partitions are a bad idea, because you should just anticipate your needs in advance and use statically sized partitions. Or dynamically changing CPU clock frequencies are a bad idea, because you should just anticipate your CPU needs and set your clock frequency in advance. Or dynamically changing process counts that adapt to different multi-core/CPU availability factors are a bad idea... you get the picture.
The idea that some computational factor can be automatically dynamically adjusted isn't necessarily a bad idea, it's just the implementation that might be.
I can summarize this article in one sentence:
"X is only useful for those who are too lazy to do Y."
It's been said about assembly language, high-level languages, garbage collection, plug-n-play, and practically any other technology you can name. It is not actually a valid criticism.
If you mod me Overrated, you are admitting that you have no penis.
On the surface auto-scaling is obviously a great thing. But it doesn't take much thought to start punching holes in it.
Lets first look at the Data center that provides such a glorious capability.
1. It is their own best interest for you to scale up. Scale up processing, disk, bandwidth or what ever. For the simple reason it's more money. Since you signed the contract you will probably be scaled well and truly before you know it. Usually you only find out when the bill comes in.
2. The data center has very little incentive to make sure you are notified in a timely manor of autoscaling. As a matter of fact this feature is usually crippled or even broken. I don't care what the contract says. The datacenter rarely honors this part of the contract to anyones satisfaction.
Now lets look at the client and the horrible things that can go wrong. By no means even remotely a complete list.
The new version of the app list.
1. Bob the developer forgets to index that new DB table. Database goes nuts trying to do a simple select. BAM autoscaling of DB CPU resources goes through the roof.
2. New AJAX call is not properly tested. For some reason it now triggers div refreshes as the mouse moves. App server is now flooded. BAM Band Width and CPU autoscale through the roof.
3. App no longer properly caches that all important query. BAM again DB and APP CPU skyrockets.
4. The genius in dev decides to make the jsession stateful. Works fine on the desktop. Works fine when load test hammers only 10 users. Oh Oh real world kicks in, in Prod. We have 10k users. Everything goes through the roof.
5. The list of new version issues goes on.
The bad guys come a knocking.
OK so now your a hot property on the net and you sign up for autoscaling so that you don't have to worry about capacity planning. You are focused on that cash machine that is your cool app.
1. You didn't know about that monster hole in the app. The bad guys inject a phishing site onto your Uber site. The phishing site is wildly successful. Oh crap we just paid for the biggest fraud site on the net.
2. The dev team leaves that back door on the site so they can maintain it remotely. Oh Oh all of a sudden you notice port 25 traffic is off the charts from the site. OMG we just uploaded 25Tbtyes in the last 24hours. You have just joined the ranks of the largest SPAM generators on the planet. You have a monster bandwidth bill and a very expensive legal bill.
3. What are these very large globs in the database all of a sudden. OH crap we left a hole and are vulnerable to SQL injection. OH crap it's all encrypted kiddie porn. Bills for bandwidth, disk and legal come a knocking.
I do have experience with this sort of thing. And it always goes sour at some point. The techies are always overruled by the marketing and business types on this. As the deal is always so great on paper. At some point something will go wrong. Software is never perfect. Between defects and bad guys you are a sitting duck for the big man carrying the bill to your door. It's only ever a matter of time.
Oh and lastly. Geuss what some times the autoscaling fails. Make that a lot of the time. And you are then off the air.
The best situation is for you as a customer of scaling is to have a close relationship with the supplier. Once you start to reach certain predefined levels of usage they should contact you and give you the option of an upgrade. Make the scaling feature by human choice. Never let the supplier decide that for you.
When your revenues scale with the services rendered, it *does* make business sense to auto-scale. Auto-scaling is a technical solution, not a business one. Being Slashdotted isn't typically associated with more commercial activity, it's associated with "hit-and-run" visitors. The same with social networks. Does Twitter even have a business model? But wherever there's a business model where margins are relatively stable but activity rises and falls, auto-scaling makes you money rather than costing you severely. Like many things, it's a tool which should be used wisely, where not paying attention can leave you missing fingers.
500GB of disk, 5TB of transfer, $5.95/mo
I have. My company lives (or dies) by the !@# SLA.
Our agreements require no less than 99.9% uptime, about 8 hours of downtime per year. We never gotten close to that - our worst year was about 2.5 hours of downtime because of a power failure at our "fully redundant" hosting facility.
In this world, where I have up to 8 hours per year, 10 minute response would be a god-send. We've just spent *alot* of money revamping our primary cluster so that we now operate with 100% full redundancy on everything. Redundant network feeds. Redundant logic servers. Redundant load balancers. Redundant database servers. All with auto failure, dynamic routing with DNS. (which is, itself, very failure tolerant)
But an application has to be constructed in a very particular way in order to scale, particularly if data integrity is important. (EG: ACID compliance SQL) This is often counter-intuitive and non-obvious, and porting an existing application to such an environment is not a quick investment. It's very typical to give up raw performance for performance scalability. We've devoted approximately 6 man-months over the past year to take full advantage of clustered, redundant computing in order to try for 1 hour over the next year along with near-linear scalability.
It's not just about capacity - it's about keeping all those !@# servers organized and coordinated!
Bottom line? Take a look at your SLA.
In our case, if we suffered a few hours of downtime every year or so, it would be an inconvenience to our users and clients. In any event, our uptime is best-of-breed in our niche-ish industry, but I'd put our uptime as mid range for hosted products overall, when you include companies that are much bigger than our still-somewhat-small rapidly-maturing startup.
Spend money where it counts. This requires an understanding of your economic base. If somebody slashdots your site, is that your golden opportunity, or is that an annoyance. In our case, a few hours of downtime if we got slashdotted wouldn't cause any particular long-term problem if it brought us down. If you have a few hundred customers paying $10/month for some cheap-o websites, a few hours of downtime every year or two won't cause much problem.
I have no problem with your religion until you decide it's reason to deprive others of the truth.
This reminds me of a large company which outsourced enthusiastically, until at one point they discovered they'd outsourced decisions about maintenance... causing the outsourcer to have control over the maintenance budget.
As you might expect, after it ballooned, they started in-sourcing!
Giving others control over financial decisions is almost always unwise, even if doing so is the newest, coolest idea of the week.
--dave
davecb@spamcop.net