Slashdot Mirror
Handling Flash Crowds From Your Garage
slashdotmsiriv writes "This paper from Microsoft Research describes the issues and tradeoffs a typical garage innovator encounters when building low-cost, scalable Internet services. The paper is a more formal analysis of the problems encountered and solutions employed a few months back when Animoto, with its new Facebook app, had to scale by a factor of 10 in 3 days. In addition, the article offers an overview of the current state of utility computing (S3, EC2, etc.) and of the most common strategies for building scalable Internet services."
Here I was picturing a bunch of people showing up in your garage for seemingly no reason. Still interesting to see how they handled the massive increase!
"Our innovator may get only one shot at widespread publicity. If and when that happens, tens of thousands of people will visit her site. But a flash crowd is notoriously fickle; "
The "researchers" offer a strange view of how the market works. If the idea is good then surely the site will enjoy numerous opportunities for growth and referral every time a happy user recommends it to a friend. A good, innovative idea will not be sunk by one underprovisioned flash crowd.
Doesn't Microsoft employ "bloggers" to seed pro MS babble to Web sites like Slashdot? Just sayin'...
If you're going to troll, it might be a good idea RTFA beforehand so that you don't make a fool of yourself. Two examples:
- The web service is implemented in Python and currently deployed on two virtual machines at Amazon EC2.
- Like Asirra, we implemented Inkblot in Python.
If they're astroturfing they aren't very good at it.
The article has very little Microsoft-specific details in it. It's basically a short explanation of high-performance content delivery and a few stories about MS Research (link because they have some cool stuff) projects and how they fared with high load traffic surging (aka Slashdotting). They specifically mention getting Slashdotted several times, as well as surviving a DDoS.
Overall I thought it was an interesting article. I didn't realize Amazon's S3 service was so inexpensive or available to "budget" sites.
"What do you despise? By this are you truly known." --Princess Irulan, Manual of Muad'Dib
/)
A paper on how to avoid slashdotting, posted on slashdot. /me clicks obsessively on links
Pet peeve: Profane people propagating perfunctory pedantry.
Well, you have to compare apples to apples with hosting, though. With Amazon's cheapest EC2 instance, you are looking at $72/month cost, or about $850/year. You have a bit more for storage, static IP, etc. But, it sure as shit beats the heck out of other boxes you could get at guys like GoDaddy and 1and1, where you get a shared box, with minimal control and cpanel or something at best, unlike root access at Amazon or admin access at GoGrid's windows boxes. You could go to services like Linode and get boxes with root access, but when you do the math, you will be hard pressed to get a comparable box at this rate. That, coupled with the flexibility of a pay-as-you-go model really does make this ideal for several situations. With the recent addition of persistent storage, you can even run full-blown db-driven apps here (something that was a pain in the ass before PS, because you had to use s3 as your permanent store). If you go to guys like Rackspace or other reputable providers, you are looking at $600-700 a MONTH as a start. Should you choose to own the iron yourself, you can probably get comparable numbers up to a certain threshold, but then you are left with hardware management issues. Of course, not that Amazon ever goes down ;)
The problem with most sites is that they never expected the sudden burst in popularity, and having never bothered to test if their service was scalable, had to rush and fix it before people start noticing reliability problems. If you at least attempted to write a scalable app to start with, even if you don't have the details nailed down, you've taken a step in the right direction.
Animoto looks like an interesting case because it's a really resource-hungry app that has to put together a video with effects and music. Most sites have trouble just serving up dynamically-updated text. All those EC2 instances and the high-bandwidth needed sounds like a lot of money. Scaling up a business plan is at least as difficult as scaling software.
if you really have one CPU unit running there non-stop -for year, it is pretty expensive, around 700 dollars I believe.
That's pretty damn cheap. A dedicated rack server is upwards of $300/month most places, and it does not provide the "elastic" part of the Amazon cloud for when your service takes on heavy demand. Rackspace, for example, provides a comparable unit at $383/mo.
You might be talking about a Virtual Private Server--there are a number of services offering similar specs in the $120-200 range...still more expensive, but more comparable to EC2.
can trace their success to that one weekend or month or season where things really took off
additionally, you misunderstand that the flash crowd is not something that comes and goes, but something that comes and stays
but sure, you are correct: a good innovative idea will find a way regardless of inability to scale quickly. some other guy will make work what you can't. you could retard your growth for awhile while you tinker with how to scale. but if some other guy takes your good, innovative idea and runs with it further and faster than you do, you are doomed to obscurity while he reaps the benefits of your good idea
so you shouldn't be giving advice on how the market works, because the fickleness you dismiss really is a big deal and is not to be taken lightly
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
http://www.usenix.org/events/usenix08/tech/full_papers/elson/elson.pdf
nosig today