Amazon, MS, Google Clouds Flop In Stress Tests

Eponymous writes "A seven month study by academics at the University of New South Wales has found that the response times of cloud compute services of Amazon, Google and Microsoft can vary by a factor of twenty depending on the time of day services are accessed. One of the lead researchers behind the stress tests reports that Amazon's EC2, Google's AppLogic and Microsoft's Azure cloud services have limitations in terms of data processing windows, response times and a lack of monitoring and reporting tools."

Comments

what the fuck is services engineering?

[Desler]

Anna Liu, Associate Professor in services engineering at the UNSW School of Computer Science told iTnews she was excited by Cloud Computing as it could potentially enable organisations to "outsource a certain amount of their risks and costs and tap into new economies of scale."

Sounds more like she has a degree in buzzword engineering.

Re:what the fuck is services engineering?

[julesh]

Well, Microsoft's Azure is in .NET, and Google's AppEngine is Python, but Amazon's EC2 is basically a virtual machine (you load your image in from S3, can be Linux or Windows). I would assume you could just write a common object in Python, have a IronPython hook to Azure, a plain Python hook to AppEngine, and a hook to whatever method you use to host your service in EC2 (like mod_python or whatever, if you're using Apache).

You could do that, but if your intent is to get as much processing power out of each platform as you can (e.g. you want to benchmark them to see how they compare with each other), you'd want to use a compiled language for your EC2 version, and probably C# for your Azure version. But you're stuck with Python for AppEngine, so you're going to be doing at least 2 and probably 3 versions. Otherwise any conclusions you make are going to be unreasonably favourable to AppEngine, as you're intentionally crippling your other systems to bring them down to the same level.

Cloud Computing? Why?

[Ralph Spoilsport]

I still don't get it. Terabyte drives cost as much as my bi-weekly beer budget, and less every day. Computing power is off of Moore's Law, but is still increasing with multicore and multiprocessors. My computer doesn't have to be hooked up to the interweb to work, nor does it require a subscription to some website to keep rolling. If I want access to the web, I can get it, but that's only a few times a day when I need it.

So, what exactly does "cloud computing" bring to the table for me?

Not much as far as I can see, other than a new crop of buzzwords.

Re:Cloud Computing? Why?

[Desler]

Well, for me, it is for low cost hosting for Django Applications, but for business it can be really interesting as there is somebody taking care of the infrastructure while they only need to care about the application itself.

So what exactly is supposed to be new about that? There have been companies providing exactly such services as that for decades.

Re:Cloud Computing? Why?

[moon3]

Cloud = "Hosting for the noobs"

The provider is managing everything for you automatically, the Cloud service takes care of pretty much everything including security so it is manageable even for non-technical dudes.

Re:Cloud Computing? Why?

[thisnamestoolong]

For the most part, I agree. I can certainly see the benefit in using remote processing capabilities (I really hate buzzwords) for things like smartphones, as it enables the user to tap into a far greater amount of processing power than could be crammed into a little handheld. For the home, however, I have a hard time imagining that it is more feasible to do your computing through the network rather than doing it locally. What about things like audio editors and games, that require latencies in the low milliseconds to be usable? Maybe we can provide that sort of speed in the future, but common sense tells me that these sorts of things will always run better off of a local machine. The processing power and hard drive space needs to be payed for one way or another, and I for one would rather pay more up front and own my hardware, rather than rent access.

Re:Cloud Computing? Why?

[Anonymous Coward]

oh yeah, because if you are a huge service provider, say 500 cloud server, and you have a spike of traffic (at least 50%) they give you 250 servers already replicated and working in one day, using the replication fairy?

Re:Cloud Computing? Why?

[Attila Dimedici]

During peak hours you can spawn a few more servers and at night you can shut down a few without having to worry about the physical hardware and their associated maintenance burden.

Right, but what does it cost, because guess what, just about everybody else wants more servers at peak hours and wants to shut down a few at night. What does the "cloud" service provider do with all of those servers when nobody wants them? How do they cover their maintenance costs for the time when their servers are idle?
That's right, by charging more for them when you want to use them. The big problem with the cloud concept is that it assumes that the need for servers is spread out evenly across the day and the year. The fact of the matter is that it isn't, most businesses need/want more servers at the same time.

Re:Cloud Computing? Why?

[jarocho]

At this stage, you the individual don't benefit tremendously from cloud computing. But your company, at *almost* any head count, might be able to leverage what's also known as utility computing today. Depending on what it does or doesn't want to bother hosting internally.

Hosted Microsoft Exchange is a concrete example of a cloud (cloud-like) service that's been gaining ground for a while now.

Wired had a read-worthy piece on Azure's principal architect Ray Ozzie last year, Ray Ozzie Wants to Push Microsoft Back Into Startup Mode. Hyperbole aside, anyone who's directly interfacing with Microsoft sales people and engineers these days will tell you, Azure is a big part of Microsoft's next money grab.

However, it's amusing that the definition of "cloud computing" continues to mean different things to different vendors, as evidenced by Amazon, Google and Microsoft offering fairly distinct and non-overlapping services. Until they come into direct competition with one another, I think this is going to continue to be seen as a novelty by many CTOs and IT decision-makers.

Is cloud computing the future? I don't know, but I think it's safe to say it's *a* future. Even if it isn't yours. :)

Re:Cloud Computing? Why?

[segedunum]

People who ask this have generally never hosted anything major before. The attraction is that it decouples your applications and server instances from real hardware and even from the specific virtualisation platform you would otherwise be sitting on. This means that a hardware failure will certainly not affect you in the same way and neither will a failure in a comparable virtualisation platform. It's on a completely different scale, and certainly with Amazon you can spread yourself across different geographic locations. I've seen many Xen VPS platforms have to be rebooted periodically for things like kernel updates and if you're dealing with real hardware then you start getting into failover and drdb, which is far too much of a pain for most development companies to worry about. You just want to host your applications somewhere. Trust me. You start worrying about this stuff very quickly otherwise.

Additionally, what makes it a 'cloud' and not just a vanilla virtualisation platform is that your storage itself is then decoupled from your machine instances themselves, as well as the hardware, in an easy way without having to faff about with clustered storage set ups yourself or through a hosting company. This makes your machine instances easily disposable and allows for pretty easy recreation of production environments as a failover or for testing and development.

Essentially, that's what's attractive about it in layman's terms. It makes it far cheaper and far less hassle to get hardware and storage redundancy when you start having to worry about it, but large companies are not going to be outsourcing their critical stuff off site with it. That's just insane. It's just a pity the whole thing has become filled full of shit by people who don't know what they're talking about like that Services Engineering nutcase in the article who is probably being paid way too much money. The article doesn't even tell you what limitations they found in any detail.

Re:Cloud Computing? Why?

[Lord Ender]

I'm developing a JRuby app for Google App Engine. I'm doing it because as a lone developer, I don't have to worry about anything but my code. I will never have to wake up to troubleshoot a network problem, OS issue, Apache oddity. I won't have to hire networking, DBA, or systems administration staff. And if my app hits off big, I won't have to re-engineer anything to make it scale. It will scale automatically.

I've played the role of network engineer, DBA, and sysadmin in the past. Now I can focus on my application.

That said, appengine is certainly not for all sorts of apps. It only supports a subset of SQL (no joins), I'm sure it won't meet the requirements for payment card processing or anything like that, and my APIs are limited. But for a good chunk of web apps, developing for the google cloud has huge advantages.

Re:Cloud Computing? Why?

[gtbritishskull]

The fact of the matter is that it isn't, most businesses need/want more servers at the same time.

Do you have a citation for that? I would think that there would be a lot of different services which need servers at different times. Most business services would peak during the day, but I would think most consumer based services (entertainment, shopping) would peak in the evening. And then you have to consider that there are other countries in the world and their day is different than yours. So, their peak times would probably be different. I am not saying that cloud computing is the way to go, but there are definitely the potential for a much better server utilization with it. And, the result will probably be that there will be time based pricing, with peak times costing more. But, it will still be more cost effective because they will still be making money on the non-peak times when individual servers would normally be idle. Also, services that would normally run at peak time, but don't need to, would be able to take advantage of the cheaper non-peak times. This is how a market works. Scarcity of resources results in efficiency. And the overall cost of the system decreases.

Re:Cloud Computing? Why?

[vertinox]

If I want access to the web, I can get it, but that's only a few times a day when I need it.

I don't know about you but I need internet access 24/7.

Secondly, cloud computer is not for nerds.

Its for non-tech types who want to outsource things.

Actually "cloud computing" is a euphemism for "outsourcing".

Well I suppose its for nerds if you are the administrator of a "cloud" but for end users not so much.

Re:Quite interesting, actually

[Anonymous Coward]

IMO the entire cloud thing is nothing more than a hype. Noone ever got asked if he wanted to have all apps running as webservices. Google, MS and others just race each other without really having a look whether the customers will buy. And i don't even want to think of the bad choice of standards they base their services on...

No Tools?

[Thomas Charron]

Google AppEngine has data reporting to a ridiculous level. This article doesn't even publish any REAL data.

I really HATE commercicles, small articles which make a claim, and then say, 'stay tuned!'.

Someone fire the author. The last paragraph reads:

"Liu will present the findings and offer developers advice on how to build robust applications to withstand the cloud's limitations at the Australian Architecture Forum in Sydney on Monday, August 24."

Wow, I at least they admit that this article has no REAL data in it, and THAT data will be released on Monday.

Re:I'm Shocked! SHOCKED!

[xplinuxmac]

Of course this is not surprising if you assume most of its users are in same timezone and do their work between 9-17. Clouds only work if the work of its clients is distributed over time, you can then aggregate dedicated resources for tasks. A cloud can not (never) properly deal with socalled peak load without making sufficient investments into hardware. Peak load occurs when everybody start using the system at the same time for intensive processing or data transactions. This is more likely to occur if your users are in the same timezone. So if your are looking for a 'good' cloud I suggest also looking at the user list. If the users are sufficiently distributed over different time zones it might scale better (this is of course not the only criteria for a good cloud). I myself prefer 'Thunder clouds' :)

Re:I'm Shocked! SHOCKED!

[jabjoe]

I can't help thinking this is just thin-client + mainframe again, and just like every other time the model has come around, it's being pushed as the future.

Re:Wave?

[Anonymous Coward]

The challenges for Wave don't rely on nearly the same challenges. Wave involves ONLY data transfer, not processing, storage, etc.. It's a protocol.

Making the comparison you've made is the same thing as saying HTTP is flawed becouse Joes Web Shack servers are slow.

Doesn't wave create thumbnails for images posted in a conversation?
Sounds like processing and storage to me...

Re:Incompetent testing, stupid article

[Beezlebub33]

In addition, there is absolutely no data presented in the article. When you say that there were problems, you should quantify the problems. How long was the response time. The article says

Response times on the service also varied by a factor of twenty depending on the time of day the services were accessed, she said.

Ok, so give me a friggin' number! Did it go from 1 min to 20 minutes? Or from 1 sec to 20 sec. Or 1 hr to 20 hrs? When did you experience these response times? Give me a graph showing the response time as function of time of day and day of week.

I am learning to hate articles that give you a little bit of information and leave out the important data. If Ms. Liu hasn't released the data, then the article should not have been written. Or she should provide it on her web page. Or provide a link to some journal where it's being published. This whole thing stinks of spin and MS FUD.

Re:Quite interesting, actually

[timeOday]

It just depends on the application. Webmail is an obvious success; why have every company re-creating this capability when everybody needs and wants the same thing? There is the issue of trust, but then, we do put our money in banks, so that is solvable.

Re:I'm Shocked! SHOCKED!

[TheRaven64]

Not really. A mainframe gives you:

• Control over your data.
• Very high availability with multiple redundancy at every level and hot-swappable parts. Most modern mainframes can have a CPU fail in the middle of a job and no one notice except the operator who is paged to plug in a new CPU if he wants the machine to return to full capacity.
• Insane I/O throughput rates, with dedicated I/O controllers so you can keep the CPU saturated with data at all times.
• A big fat support contract which lets you wake someone up in the middle of the night and make them fix your problems.

I don't really see how this is similar to the cloud at all.

Confusing terms

[SanityInAnarchy]

You're confusing two definitions of "cloud".

One is the idea of putting everything into a webservice. The other is the idea of utility computing. They often overlap, but plenty of web services run their own datacenters, and there are plenty of applications of utility computing beyond web services.

Specifically, your "scalability issues" are relevant to the "utility computing" part, but not so much to the "web services" part -- unless you were bringing up issues completely irrelevant to this article.

This is my main annoyance with the use of the word "cloud" -- even people with some technical knowledge still get fooled into thinking one kind of "cloud" has anything at all to do with another type of cloud.