Amazon, MS, Google Clouds Flop In Stress Tests

Eponymous writes "A seven month study by academics at the University of New South Wales has found that the response times of cloud compute services of Amazon, Google and Microsoft can vary by a factor of twenty depending on the time of day services are accessed. One of the lead researchers behind the stress tests reports that Amazon's EC2, Google's AppLogic and Microsoft's Azure cloud services have limitations in terms of data processing windows, response times and a lack of monitoring and reporting tools."

Comments

First!

[Anonymous Coward]

Cloud free and lightning fast!

what the fuck is services engineering?

[Desler]

Anna Liu, Associate Professor in services engineering at the UNSW School of Computer Science told iTnews she was excited by Cloud Computing as it could potentially enable organisations to "outsource a certain amount of their risks and costs and tap into new economies of scale."

Sounds more like she has a degree in buzzword engineering.

Re:what the fuck is services engineering?

[eldavojohn]

Anna Liu, Associate Professor in services engineering at the UNSW School of Computer Science told iTnews she was excited by Cloud Computing as it could potentially enable organisations to "outsource a certain amount of their risks and costs and tap into new economies of scale."

Sounds more like she has a degree in buzzword engineering.

From her homepage at UNSW, it seems to be the creation and study of services but her focus seems to be on cloud computing with the "services" being concentrated on these subjects. While a lot of her about page seems to be buzzwords and journal writing, I really wish they would release their "interoperable service software" and would be interested in seeing their final report for more specific metrics. Her blog doesn't say much about it. I'd give her the benefit of the doubt, she says in the article, "We saw a lot of hype and confusion, and decided to lead a team of researchers and actually get our hands dirty with this stuff." She also said:

Using Google AppEngine, none of your data processing tasks can last any longer than thirty seconds, or it throws an exception back at you. This is very consistent with the Google business model - they want to enable simple web applications to thrive on the Internet. AppEngine is there to enable the rapid development of simple web applications that don't include intense compute at the back end. - Anna Liu

Which I found interesting. Again, kind of hard to judge the merits behind this research without even a brief description of what the services were ... a singular value decomposition service? A return huge data sets from a database table service? A prime factorization service? A file intensive I/O service? I'm also curious as to what hoops one has to jump through to get those interoperable across all three systems ... after all Microsoft is just .NET, right? Is this rewriting something 3 times or making shared objects or what?

Wave?

[tygerstripes]

I wonder what the implications will be for Wave? Real-time updates across multiple servers present very similar challenges to cloud-computing. If the relevant protocols have the same problems then it raises doubts over the scalability of the Wave protocol.

Re:Wave?

[Thomas Charron]

The challenges for Wave don't rely on nearly the same challenges. Wave involves ONLY data transfer, not processing, storage, etc.. It's a protocol.

Making the comparison you've made is the same thing as saying HTTP is flawed becouse Joes Web Shack servers are slow.

Cloud Computing? Why?

[Ralph Spoilsport]

I still don't get it. Terabyte drives cost as much as my bi-weekly beer budget, and less every day. Computing power is off of Moore's Law, but is still increasing with multicore and multiprocessors. My computer doesn't have to be hooked up to the interweb to work, nor does it require a subscription to some website to keep rolling. If I want access to the web, I can get it, but that's only a few times a day when I need it.

So, what exactly does "cloud computing" bring to the table for me?

Not much as far as I can see, other than a new crop of buzzwords.

Re:Cloud Computing? Why?

[FooBarWidget]

Cloud computing has the benefit that when you need to expand your server park's capacity, you don't have to wait for several weeks for the hardware vendor to deliver the hardware. Instead you outsource that job to the cloud vendor. You can more quickly respond to both increase and decrease in traffic. During peak hours you can spawn a few more servers and at night you can shut down a few without having to worry about the physical hardware and their associated maintenance burden.

Re:Cloud Computing? Why?

[moon3]

Cloud = "Hosting for the noobs"

The provider is managing everything for you automatically, the Cloud service takes care of pretty much everything including security so it is manageable even for non-technical dudes.

Re:Cloud Computing? Why?

[Sockatume]

Essentially, it allows you to treat any net-connected computer as a dumb terminal with web services acting as your actual computer.

Re:Cloud Computing? Why?

[segedunum]

People who ask this have generally never hosted anything major before. The attraction is that it decouples your applications and server instances from real hardware and even from the specific virtualisation platform you would otherwise be sitting on. This means that a hardware failure will certainly not affect you in the same way and neither will a failure in a comparable virtualisation platform. It's on a completely different scale, and certainly with Amazon you can spread yourself across different geographic locations. I've seen many Xen VPS platforms have to be rebooted periodically for things like kernel updates and if you're dealing with real hardware then you start getting into failover and drdb, which is far too much of a pain for most development companies to worry about. You just want to host your applications somewhere. Trust me. You start worrying about this stuff very quickly otherwise.

Additionally, what makes it a 'cloud' and not just a vanilla virtualisation platform is that your storage itself is then decoupled from your machine instances themselves, as well as the hardware, in an easy way without having to faff about with clustered storage set ups yourself or through a hosting company. This makes your machine instances easily disposable and allows for pretty easy recreation of production environments as a failover or for testing and development.

Essentially, that's what's attractive about it in layman's terms. It makes it far cheaper and far less hassle to get hardware and storage redundancy when you start having to worry about it, but large companies are not going to be outsourcing their critical stuff off site with it. That's just insane. It's just a pity the whole thing has become filled full of shit by people who don't know what they're talking about like that Services Engineering nutcase in the article who is probably being paid way too much money. The article doesn't even tell you what limitations they found in any detail.

Re:Cloud Computing? Why?

[Lord Ender]

I'm developing a JRuby app for Google App Engine. I'm doing it because as a lone developer, I don't have to worry about anything but my code. I will never have to wake up to troubleshoot a network problem, OS issue, Apache oddity. I won't have to hire networking, DBA, or systems administration staff. And if my app hits off big, I won't have to re-engineer anything to make it scale. It will scale automatically.

I've played the role of network engineer, DBA, and sysadmin in the past. Now I can focus on my application.

That said, appengine is certainly not for all sorts of apps. It only supports a subset of SQL (no joins), I'm sure it won't meet the requirements for payment card processing or anything like that, and my APIs are limited. But for a good chunk of web apps, developing for the google cloud has huge advantages.

Re:Quite interesting, actually

[Anonymous Coward]

IMO the entire cloud thing is nothing more than a hype. Noone ever got asked if he wanted to have all apps running as webservices. Google, MS and others just race each other without really having a look whether the customers will buy. And i don't even want to think of the bad choice of standards they base their services on...

No Tools?

[Thomas Charron]

Google AppEngine has data reporting to a ridiculous level. This article doesn't even publish any REAL data.

I really HATE commercicles, small articles which make a claim, and then say, 'stay tuned!'.

Someone fire the author. The last paragraph reads:

"Liu will present the findings and offer developers advice on how to build robust applications to withstand the cloud's limitations at the Australian Architecture Forum in Sydney on Monday, August 24."

Wow, I at least they admit that this article has no REAL data in it, and THAT data will be released on Monday.

AppLogic?

[Roguelazer]

Buh? As far as I can tell, Google doesn't have a platform called "AppLogic". Perhaps they were referring to App Engine? And it's not even the editors' fault this time -- TFA has the terms wrong too. That really inspires confidence...

Re:I'm Shocked! SHOCKED!

[jabjoe]

I can't help thinking this is just thin-client + mainframe again, and just like every other time the model has come around, it's being pushed as the future.

Re:Quite interesting, actually

[lenehey]

I dunno. If someone steals my money, it can be replaced by the bank whose security system failed. If someone steals my secrets, there is no remedy.

Re:I'm Shocked! SHOCKED!

[TheRaven64]

Not really. A mainframe gives you:

• Control over your data.
• Very high availability with multiple redundancy at every level and hot-swappable parts. Most modern mainframes can have a CPU fail in the middle of a job and no one notice except the operator who is paged to plug in a new CPU if he wants the machine to return to full capacity.
• Insane I/O throughput rates, with dedicated I/O controllers so you can keep the CPU saturated with data at all times.
• A big fat support contract which lets you wake someone up in the middle of the night and make them fix your problems.

I don't really see how this is similar to the cloud at all.

Confusing terms

[SanityInAnarchy]

You're confusing two definitions of "cloud".

One is the idea of putting everything into a webservice. The other is the idea of utility computing. They often overlap, but plenty of web services run their own datacenters, and there are plenty of applications of utility computing beyond web services.

Specifically, your "scalability issues" are relevant to the "utility computing" part, but not so much to the "web services" part -- unless you were bringing up issues completely irrelevant to this article.

This is my main annoyance with the use of the word "cloud" -- even people with some technical knowledge still get fooled into thinking one kind of "cloud" has anything at all to do with another type of cloud.