Handling Flash Crowds From Your Garage

slashdotmsiriv writes "This paper from Microsoft Research describes the issues and tradeoffs a typical garage innovator encounters when building low-cost, scalable Internet services. The paper is a more formal analysis of the problems encountered and solutions employed a few months back when Animoto, with its new Facebook app, had to scale by a factor of 10 in 3 days. In addition, the article offers an overview of the current state of utility computing (S3, EC2, etc.) and of the most common strategies for building scalable Internet services."

Comments

Not Flash Mobs?

[penguin king]

Here I was picturing a bunch of people showing up in your garage for seemingly no reason. Still interesting to see how they handled the massive increase!

Re:Not Flash Mobs?

[RuBLed]

And here I was thinking that it was about handling a crowd flashing for you in front of your garage...

Re:Not Flash Mobs?

[dredwerker]

I thought it was something to with actionscript.

Re:Not Flash Mobs?

[ericvids]

Question: So how DO you handle flash crowds?

MS Answer: We don't. We'll force them to install Silverlight.

Re:Not Flash Mobs?

[houghi]

And here I was thinking that it was about handling a crowd flashing for you in front of your garage...

You must be non-American, because in Soviet America you flash the crowd.

Re:Not Flash Mobs?

[Divebus]

When I read "This paper from Microsoft Research..." I thought... oh yeah, they're in Cupertino.

Re:Not Flash Mobs?

[MJMullinII]

Nah, that's easy.

TRUST ME, they scatter pretty easy when wheel into your driveway at 40 mph.

The few that remain can easily be cleaned up with a fire/water hose.

Re:Not Flash Mobs?

[wstfgl]

... a fire/water hose.

Your ideas are intriguing to me and I wish to subscribe to your newsletter.

The method:

[Ethanol-fueled]

1) Ride on myspace or facebook's coattails
2) ???
3) Profit!

Re:The method:

[pimpimpim]

Actually, in this case I really wonder if he manages to make a profit. I looked at the amazon cloud recently, and if you really have one CPU unit running there non-stop -for year, it is pretty expensive, around 700 dollars I believe. For something like that you could also just get a dedicated rack server. So if this guy apparently needs a lot more than that, he sure needs to make quite a lot of money from his facebook app, because he will be spending several thousands of dollars on CPU power.

Re:The method:

[Ethanol-fueled]

Hmm, you're right. Better revamp the method:

1) Be myspace or facebook
2) ride on the coattails of self-absorbed attention whores
3) ???
4) Profit!


5) throw doggie bones to javascript noobs so that sites' users could be spammed with even more movie quizzes.(You too can be a myspace developer! Valid e-mail address and fake phone number required)

Re:The method:

[ruphus13]

Well, you have to compare apples to apples with hosting, though. With Amazon's cheapest EC2 instance, you are looking at $72/month cost, or about $850/year. You have a bit more for storage, static IP, etc. But, it sure as shit beats the heck out of other boxes you could get at guys like GoDaddy and 1and1, where you get a shared box, with minimal control and cpanel or something at best, unlike root access at Amazon or admin access at GoGrid's windows boxes. You could go to services like Linode and get boxes with root access, but when you do the math, you will be hard pressed to get a comparable box at this rate. That, coupled with the flexibility of a pay-as-you-go model really does make this ideal for several situations. With the recent addition of persistent storage, you can even run full-blown db-driven apps here (something that was a pain in the ass before PS, because you had to use s3 as your permanent store). If you go to guys like Rackspace or other reputable providers, you are looking at $600-700 a MONTH as a start. Should you choose to own the iron yourself, you can probably get comparable numbers up to a certain threshold, but then you are left with hardware management issues. Of course, not that Amazon ever goes down ;)

Re:The method:

[pimpimpim]

hmmhmm, no no. I have a virtual server with root access and even ability to use serial access over ssh if everything else fails. This is about 9 euro per month. for 72 euro per month I can get a nice single core machine with 2 GB ram and 2X250 GB in raid 1. And I'm not even at the cheapest provider. Thing is of course, that such providers work with yearly contracts and you cannot catch peak usage as nicely as with amazon's cloud. I am just saying that when you as a garage shop get a large amount of traffic at once, you not only need the hardware, you still need to make sure you make a considerable amount of cash before you bankrupt yourself into popularity.

Re:The method:

[mr_matticus]

if you really have one CPU unit running there non-stop -for year, it is pretty expensive, around 700 dollars I believe.

That's pretty damn cheap. A dedicated rack server is upwards of $300/month most places, and it does not provide the "elastic" part of the Amazon cloud for when your service takes on heavy demand. Rackspace, for example, provides a comparable unit at $383/mo.

You might be talking about a Virtual Private Server--there are a number of services offering similar specs in the $120-200 range...still more expensive, but more comparable to EC2.

Re:The method:

[Peeteriz]

Well, that was the whole point. One CPU-unit for the whole year is more expensive than needed; but for such cases you need 0.01 'units' non-stop for the whole year, and 100 units for one weekend, it's much cheaper in Amazon's way.

Re:The method:

[Anonymous Coward]

Absolutely agree that cloud hosting services offer significant economies over traditional hosting. While we're naming vendors, a more complete list of cloud vendors includes the following (and most offer a much fuller range of web hosting services than EC2!):

US: Amazon EC2, MediaTemple, GoGrid, Mosso, Linode, Joyent

UK: ElasticHosts, FlexiScale

Re:The method:

[Anonymous Coward]

linode is #20/mo.

Re:The method:

[TheLink]

But where's the money coming from?

IIRC the article mentioned thousands of servers at one point.

Re:The method:

[merreborn]

Softlayer has multi-core boxes starting at $150/month; we've got a box with them with a 15k RPM SCSI drive for about $300/mo.

For dinky personal projects, I've got a dedicated Athlon XP 2400+ with half a gig of ram with a little no-name provider -- and it only runs me $50/mo.

I've seen all sorts of prices in the $50 - $300 range for varying hardware. If you're willing to gamble on a lesser known host, you can get hardware cheap.

I wouldn't necessarily recommend running an established webapp with thousands of active users in a datacenter like this, but when you're at the "garage" stage, they're more than sufficient. They're certainly preferable to shared hosting on a grade-A provider, from what I've seen.

Re:The method:

[mr_matticus]

None of that is comparable hardware. Softlayer has pretty good prices, but you're still looking at $200/mo with no elasticity for the best approximation. EC2 is still roughly 1/3 the price.

An Athlon at a no-name provider isn't even worth mentioning in comparison to one of these units. Web hosting can be found for cheap. I wouldn't even jokingly put them in the same class, though.

Re:The method:

[mr_matticus]

No, it's not. Its closest plan to an EC2 "small" instance is $80/month for one-third the storage and less RAM. If Linode offered a comparable service, it'd be about $110/month.

Re:The method:

[cheater512]

I have quite a nice VPS with 512mb ram at SliceHost.
$38/month.

Your prices are a tad high.

Re:The method:

[mr_matticus]

Your plan has less than one-third the RAM and about one-eighth the storage.

EC2 minimum units: 1.7GB RAM, 160GB storage. Price at your host for the closest analogue (2GB/80GB): $140 (well within the 120-200 range I stated). Total price to operate the EC2 unit: $72 plus bandwidth.

Moreover, where EC2 really cuts into the competition is scaling at larger units, particularly with regard to hourly billing (allowing much more granularity in service and pricing). EC2's mid-size unit absolutely crushes the specs of your host's very best plan, and at a comparable price.

Your ability to compare services is subpar. You can get webhosting VPS for $10/month with some plans. That doesn't make it comparable to an EC2 unit. That said, your host has generous bandwidth caps for the price points and is certainly adequate for someone who doesn't need the muscle of EC2 services.

Re:The method:

[mibus]

A dedicated rack server is upwards of $300/month most places

You must not have shopped around a lot! We're paying ~$160/mo for some really nice Core2Duo machines with healthy specs (2GB RAM / 160GB HDD / 2TB quota, I think).

Astro Turf

[Frosty Piss]

Doesn't Microsoft employ "bloggers" to seed pro MS babble to Web sites like Slashdot? Just sayin' ...

Re:Astro Turf

[nmb3000]

Doesn't Microsoft employ "bloggers" to seed pro MS babble to Web sites like Slashdot? Just sayin'...

If you're going to troll, it might be a good idea RTFA beforehand so that you don't make a fool of yourself. Two examples:

- The web service is implemented in Python and currently deployed on two virtual machines at Amazon EC2.
- Like Asirra, we implemented Inkblot in Python.

If they're astroturfing they aren't very good at it.

The article has very little Microsoft-specific details in it. It's basically a short explanation of high-performance content delivery and a few stories about MS Research (link because they have some cool stuff) projects and how they fared with high load traffic surging (aka Slashdotting). They specifically mention getting Slashdotted several times, as well as surviving a DDoS.

Overall I thought it was an interesting article. I didn't realize Amazon's S3 service was so inexpensive or available to "budget" sites.

Re:Astro Turf

[Maxmin]

Doesn't Microsoft employ "bloggers" to seed pro MS babble to Web sites like Slashdot? Just sayin'...

If you're going to troll, it might be a good idea RTFA beforehand so that you don't make a fool of yourself. Two examples:

Erm, for the record they also cover applications created by MS Research, using MS technologies - e.g. one called MapCruncher, which runs/ran on IIS. See 5 Application Design and Flash Crowd Experiences. Another examples include SQL Server, but the article's pretty much technology agnostic, on the whole...

We were surprised that our web server had failed to keep up with the request stream until we realized the dataset was many times larger... The next day, we published our maps to Amazon S3, and had no further performance problems.

In all, a very interesting article. I didn't get much or any MS-centric chest-thumping from it.

Re:Astro Turf

[Anonymous Coward]

Nice try. What MS group do you work for?

Re:Astro Turf

[RulerOf]

Doesn't Microsoft employ "bloggers" to seed pro MS babble to Web sites like Slashdot?

If a sizable portion of your direct competition's most enlightened (using that term loosely) users all congregated in one place for you to spread propaganda on them.... Wouldn't you?

Google?

[Anonymous Coward]

This paper from Microsoft Research describes the issues and tradeoffs a typical garage innovator encounters when building low-cost, scalable Internet services.

Anyone else initially think that Microsoft was talking about Google after reading that first sentence?

Slashdot training

[Cur8or]

Linking your site in your sig is a good way to test the scaling as the slashdot crowd hits your site.

Where are the pix?

[mcrbids]

Since I can't see any pix in the area near the bottom called "Figure: DNS servers fail over very quickly when an upstream server fails" - does that mean that the flash crowd called "SLASHDOT" has taken down this part of the article called "Handling Flash Crowds..." ?

I mean sheesh! They even mention slashdot!

Re:Where are the pix?

[damburger]

Surely nothing can trash their argument so much as an abject (and amusing) failure to put it into practice?

How can we believe in their solutions for letting small-time web services handle usage spikes when their evidently can't?

Re:Where are the pix?

[Schlage]

Considering that the paper detailing their argument, and the images/figures used therein, is hosted somewhere other than Microsoft's own website (at usenix.org, to be specific), I don't think its really accurate to say that someone else's failure to effectively handle traffic is a poor reflection on the effectiveness of their solution.

Misleading pretense

[wombatmobile]

"Our innovator may get only one shot at widespread publicity. If and when that happens, tens of thousands of people will visit her site. But a flash crowd is notoriously fickle; "

The "researchers" offer a strange view of how the market works. If the idea is good then surely the site will enjoy numerous opportunities for growth and referral every time a happy user recommends it to a friend. A good, innovative idea will not be sunk by one underprovisioned flash crowd.

Re:Misleading pretense

[spazdor]

I think you overestimate the attention span of the type of people who compulsively install Facebook apps.

Re:Misleading pretense

[Bromskloss]

I think you overestimate the attention span of the type of people who compulsively install Facebook apps.

Even if you would lose that first crowd, you might get other users at a later point.

Re:Misleading pretense

[spazdor]

The first crowd is a different class of user from the general public. It's a small subset of the Facebook usership that forwards almost everything they receive to everyone they know. Pandering to that particular crowd is a Facebook developer's foremost goal, because they are the ones who will drive exponential growth, if it's going to happen at all.

I think market research will show that this core group of irritating people are just as capricious with the "block app" button as with the "forward". So assuming your idea is good enough to spread among the primary group, your first chance is generally the only one you get.

Re:Misleading pretense

[argent]

It's a small subset of the Facebook usership that forwards almost everything they receive to everyone they know. Pandering to that particular crowd is a Facebook developer's foremost goal, because they are the ones who will drive exponential growth, if it's going to happen at all.

So basically Facebook selects for applications that are attractive to the kind of people who forward spam.

Thanks for the warning.

Re:Misleading pretense

[neuromancer23]

Agreed. To this day, MySpace still works less than 50% of the time. Of course, they are using SQLServer and Windows...

Re:Misleading pretense

[Anonymous Coward]

I work for one of the major ad networks in the app developer field (posting AC from the office). First to market is ridiculously valuable, great user experience is ridiculously important.

If you have a Facebook account, check out this games made by PlayFish: Word Challenge. It's basically a rehash of TextTwist or WordTwist, etc. There must be a dozen similar games. And yet Word Challenge rose to the top in a matter of weeks.

As a guy who's job involves playing with these things, I'll concede that Word Challenge is my favorite variant, but if they didn't have the ability to scale (check out this growth chart on the two apps), they were in deep trouble. Getting your first users in these apps is so important that developers regularly talk about strategies for generating "seed users".

When your 15 min is called you are getting your opportunity for a significant base of seed users. If you miss that opportunity, you're not done, but you've just made your job much harder.

Re:Misleading pretense

[Dragonslicer]

"Our innovator may get only one shot at widespread publicity. If and when that happens, tens of thousands of people will visit her site. But a flash crowd is notoriously fickle; "

The "researchers" offer a strange view of how the market works. If the idea is good then surely the site will enjoy numerous opportunities for growth and referral every time a happy user recommends it to a friend. A good, innovative idea will not be sunk by one underprovisioned flash crowd.

I don't see anything in that first quote about the idea being good, only popular for a short period of time.

Re:Misleading pretense

[Kalriath]

I think you've forgotten to mention the real reason Myspace is down half the time - it's ColdFusion, run by Fox.

Brand protection:

[Geak]

FTA:

including one that was (literally) Slashdotted

Anybody here think slashdot should be protecting it's brand here? Isn't this similar to using google as a verb? I think this is the only place one should be allowed to use that term. Microsoft most definitely shouldn't be allowed.

Re:Brand protection:

[drinkypoo]

Anybody here think slashdot should be protecting it's brand here?

Yes.

Isn't this similar to using google as a verb?

The canonical example is "Photoshop". It predates google considerably. HTH.

I think this is the only place one should be allowed to use that term.

I think you're ugly, and your mother dresses you funny.

Microsoft most definitely shouldn't be allowed.

Why not? We are free to refer to buying good software and ruining it, then forcing it on people through your monopoly position as "Microsofting" it.

Intellectual property laws are not handed down from god (unless they're in lost commandments 11-15.)

Re:Brand protection:

[Anonymous Coward]

The canonical example is "Photoshop". It predates google considerably. HTH.

You seem to have misspelled Xerox.

Kevin

Re:Brand protection:

[ins0m]

You seem to have misspelled Kleenex.

Re:Brand protection:

[jahudabudy]

Kleenex was never verbed.

Re:Brand protection:

[Anonymous Coward]

You seem to have misspelled Hoover, old chap.

Re:Brand protection:

[Anonymous Coward]

This one might win (certainly a better fit than Kleenex as a noun for paper tissue) but I've never heard of hoovering anything.

Perhaps it was a localism though.

Kevin

Make your application... efficient?

[Colin Smith]

Then you don't need as much brute force ?

 

SNOWED!

[Frosty Piss]

Yes, but this is all the signs of a *very good* AstroTurf, so good, you don't even realise that this is what's going on. It's "positive" Microsoft PR, and that's what couts. Oh, and it's *even better* that they used Python. But in the end, it's still MS PR babble, and you all swallowed it HOOK LINE AND SINKER.

Microsoft is learning... And learning good, they have the jaded Slashdot crowd SNOWED!

Re:SNOWED!

[thermian]

Hmm, and argument that neither requires nor accepts any proof, and it has CAPS LOCK words!

You win, sir.

As to what you win, well....

Re:SNOWED!

[Anonymous Coward]

Which MS group do you work for? Are you at the main campas or Red West?

How to handle Flash Crowds?

[Rik Sweeney]

Easy, Flashblock

Great.

[ericvids]

A paper on how to avoid slashdotting, posted on slashdot. /me clicks obsessively on links

Mixed tech history metaphor

[Mathinker]

They've mixed their metaphors, since it was the founders of Apple who innovated in a garage, and Google who provide a scalable Internet service...

What?

[Anonymous Coward]

What the hell is that article talking about? I don't understand a single flippin' word of it!

Facebook

[codeButcher]

... when Animoto, with its new Facebook app, had to scale by a factor of 10 in 3 days.

Of course, I'm completely guessing here, but they probably required you to invite 20 brazillion of your imaginary Facebook friends before you could install it.

Re:Facebook

[Dragonslicer]

... when Animoto, with its new Facebook app, had to scale by a factor of 10 in 3 days.

Of course, I'm completely guessing here, but they probably required you to invite 20 brazillion of your imaginary Facebook friends before you could install it.

How bad does an application have to be to be brought down by 20 Brazilians?

Re:Facebook

[blueZ3]

20 Brazilians is not as bad as six Spaniards, a Frenchman, and two guys from Nigeria.

Re:Facebook

[codeButcher]

.... and if I had any Brazilian FB friends, I might have bothered to try the app out too :-)

The hard part is realizing that you need to scale

[Anonymous Coward]

The problem with most sites is that they never expected the sudden burst in popularity, and having never bothered to test if their service was scalable, had to rush and fix it before people start noticing reliability problems. If you at least attempted to write a scalable app to start with, even if you don't have the details nailed down, you've taken a step in the right direction.

Animoto looks like an interesting case because it's a really resource-hungry app that has to put together a video with effects and music. Most sites have trouble just serving up dynamically-updated text. All those EC2 instances and the high-bandwidth needed sounds like a lot of money. Scaling up a business plan is at least as difficult as scaling software.

most successful sites

[circletimessquare]

can trace their success to that one weekend or month or season where things really took off

additionally, you misunderstand that the flash crowd is not something that comes and goes, but something that comes and stays

but sure, you are correct: a good innovative idea will find a way regardless of inability to scale quickly. some other guy will make work what you can't. you could retard your growth for awhile while you tinker with how to scale. but if some other guy takes your good, innovative idea and runs with it further and faster than you do, you are doomed to obscurity while he reaps the benefits of your good idea

so you shouldn't be giving advice on how the market works, because the fickleness you dismiss really is a big deal and is not to be taken lightly

And for those of us who speak English?

[gsslay]

Could someone provide a translation of the summary for those of us who speak English rather than promotional BS? .. on second thoughts, never mind.

Load Balancing

[Anonymous Coward]

Try Proto Balance: www.protonet.co.za

Good old MS at work again...

[V!NCENT]

First they use the argument that any geek/nerd with a good idea can put them out of business in their anti-trust case (argument also made in Hackers 3: Antitrust (movie)) and now they try to discourage anyone trying to set up something in their garage (while covering their asses to make you believe they try to help these people by giving them 'good advice'). This is ofcourse is totally unrelated to the fact that MS is gearing towards web 3.0 (turn on sarcasm scanners)...

Re:Good old MS at work again...

[Anonymous Coward]

Have you actually read the linked articles? The Microsoft Research paper talks about using Amazon S3 Instances, BerkelyDB, and Python. Its a paper submitted to Usenix! This blind hatred of Microsoft is just stupid. The rightscale blog post is of course promotional for themselves, since they helped provide the expansion needed by Animoto. But its still informative.

Re:Good old MS at work again...

[V!NCENT]

Apple wan't founded by two kids in a garage? Oh well... But here is a website with the quote and the source:
http://www.twokidsinagarage.com/quotes.html
But ignore Nerv because that is from the movie.

Well it's been going on forever really

[drinkypoo]

Because in English, all nouns will eventually be verbed, and vice versa. I'm sure there's some ivory tower dweller somewhere who can tell us what the first recorded example is, and that it probably wasn't in English. Xerox, however, has been superseded because only old people use photocopiers any more (except at the library sometimes. Most people don't bring in a scanner.)

WTF?

[rts008]

I handle flash crowds from my garage with pepper spray, and if that does not work, having a shotgun for escalation.*sarcasm*

In this contest to be clever with the language, all I see is a bunch of idiots outsmarting themselves.

or download the pdf

[CBravo]

http://www.usenix.org/events/usenix08/tech/full_papers/elson/elson.pdf

someone post this on reddit

[zuperduperman]

There's some irony in this appearing on slashdot today: it seems like all I can get out of reddit all morning is a "Service Unavailable" message :-)

Re:Not Flash Mobs? Easy

[infonography]

Just turn off the local displacement booths when you reach a certain threshold. Make them walk a few blocks extra that's all.

"...The next thing anyone knows, every man, woman and child in the country has decided that he wants to see the red tide at Hermosa Beach..."

"Another flash crowd. It figures," said Jerryberry. "You can get a flash crowd anywhere there are displacement booths."

From Flash Crowd, by Larry Niven.