How MySpace Generates Enough Load To Test Itself

An anonymous reader points out this article about "...how a big site like MySpace uses thousands of cloud computing cores to do performance testing on its live site. There are some really great numbers in there from the performance tests, like generating 16GB/second of bandwidth and 77,000 hits/second during testing (not including the live traffic on the site at the time)."

Comments

Wait, what?

[girlintraining]

The real news here is that people still use myspace. -_-

Re:Wait, what?

[Anonymous Coward]

That's why they have to simulate a bunch of fake people using myspace to do load testing.

Re:Wait, what?

[Anonymous Coward]

You mean the *sad* news here is that people still use myspace.

Re:Wait, what?

[Pojut]

Agreed.

Is there ANYONE on here that still uses Myspace?

Re:Wait, what?

[ircmaxell]

The real news (to me at least) is that they use 77k hits per second to TEST. Normally, when I test at least, I try to go for at least 150% to 200% expected peak load. Assuming that they aren't doing this during peak hours, that puts their expected PEAK load somewhere around 100k hits per second or so... Large, yes. Large for that size of site? No way... Using RMS, I'd say that they are averaging around 25k to 35k hits per second IF they are testing for a 100k peak... That puts MYSPACE (once one of the largest sites on the internet at one time) around an Alexa rank of about 600 (based on a few other sites I know with that load level)... Since Alexa currently rates them at 14, that implies two possibilities. Either Alexa's rank information is demographically biased, or my figures are completely out of the blue... You decide...

Re:Wait, what?

[DIplomatic]

I'm not sure where you're getting 77k hits per second. FTFA: "The goal was to test an additional 1 million concurrent users on their live site stressing the new video features. The key word here is ‘concurrent’. Not over the course of an hour or day 1 million users concurrently active on the site."

Re:Wait, what?

[ircmaxell]

FTS:

77,000 hits/second during testing

(77000 hits/second) / 1 million users = 4.6 hits/minute per user (That's a little bit higher than the figures from my experience, but still well within reason)...

Re:Wait, what?

[cgenman]

They wanted to supplement the live traffic with test traffic to get an idea of the overall performance impact of the new launch on the entire infrastructure.

So the 77k hits per second wasn't their expected peak load, but their expected delta in peak load after opening online streaming in... (looks up) New Zealand. 77k hits per second for streaming in New Zealand doesn't sound that far off.

Re:Wait, what?

[Volante3192]

They were adding 77k hits/sec to their live traffic, not testing against 77k hits/sec.

I.e., if 4 mil live users were hitting MySpace during the test, MySpace's servers were actually feeling the impact of 5 mil.

Re:Wait, what?

[pushing-robot]

That's why they have to simulate a bunch of fake people using myspace to do load testing.

To be fair, they're more intelligent than the real people using myspace.

Re:Wait, what?

[xOneca]

I think everyone here has his own web server at home.

Re:Wait, what?

[Arthur Grumbine]

I'm pretty sure this test was just for nostalgic purposes:

  MySpace Admin #1: Remember when we used to have millions of hits per hour?

  MySpace Admin #2: Yeah... those were the days. I was, like, a rockstar to my friends.

  MySpace Admin #1: Yeah, my friends thought I had the coolest job in the world, working for MySpace.

  MySpace Admin #1 and #2: *sigh*

  MySpace Admin #2:...hey - I know! We could pay some company to run a load test, it'll be just like the good ol' days!

  MySpace Admin #1: Yeah!!

Re:Wait, what?

[ircmaxell]

Correct. Based on my assumption previously that the site was at less than peak load during testing (let's call that initial load: x) here's how I calculated my figures...

x + 77k = testpercentage * peakload

Two things become apparent here. First is the assumption that x 1 (or in otherwords x + 77k > peakload). Why this assumption? Now, I assumed that they were doing capacity tests (they could be doing functionality tests, but I got the impression from the summary and TFA that it was capacity testing)... Next, let's look at normal operation x = peakload, x MUST BE LESS THAN 231k hits/sec. If x is 50% of peak load, then x must be less than 77k hits/sec... Sure, there are a lot of assumptions about these numbers, but it still serves as a rough estimation (so long as you realize what's assumed)...

Re:Wait, what?

[Anonymous Coward]

This was actually just to test if the MySpace servers were still working... after all, they haven't been used for a VERY long time.

Re:Wait, what?

[s2theg]

Not impressed. I normally generate enough load to finish before the next page even loads.

Re:Wait, what?

[BitZtream]

The funny part is that I read the summary and the article, then came and read your comment and it was the first time I realized they were talking about MySpace and not Facebook.

Funny that they are one and the same to me now.

Re:Wait, what?

[elfprince13]

I haven't deleted my account yet. I can't remember my last login though.

Re:Wait, what?

[DerekLyons]

The real news (to me at least) is that they use 77k hits per second to TEST.

According to TFA - the 'test' users were in addition to the normal users, simulating the additional load of the new services. So actual load on the production servers was something well north of 77khits/sec.

Re:Wait, what?

[Vlad_Drak]

This is testing just one part of the site (streaming video in NZ no less!), so you can't make wild generalizations based on those numbers if you expect any accuracy at all. You're not even going to get a 'rough' estimation.

For example, viewmorepics.myspace.com might do X req/s duting peak and home.myspace.com, www.myspace.com or music.myspace.com might do something wildly different because they have completely different traffic patterns.

Re:Wait, what?

[0100010001010011]

Letting laymen edit HTML always worked out for the best.

Although the Myspace Worm has to be one of the most hilarious things I've ever read.

Re:Wait, what?

[Monsieur_F]

Some artists still use it to announce their next concerts

Re:Wait, what?

[Volante3192]

From my viewpoint, which is likely more amateur armchair as it were, it just sounded like they were testing the extra load, not so much peak usage. And even then they were having to shuffle load around.

One could question the logic of trying to stress test your live servers when you're the size of MySpace as well.

Of course it could have been a third thing they were trying to accomplish as TFA was pretty weak.

Re:Wait, what?

[strawberryutopia]

MySpace still exists?

Re:Wait, what?

[ImYourVirus]

Eventful.com works well for that, you don't have to scour hundreds of your favorite bands websites every day to find out when they're coming to town, sign up, add your bands, and they will send you an email when that band books an event in your area.

I am in no way affiliated with eventful.com, just to throw that out there so no one gets confused.

Re:Wait, what?

[Anonymous Coward]

ok.

They use to mask the reality

[Anonymous Coward]

That the site only has one actual visitor anymore.

Re:They use to mask the reality

[BJ_Covert_Action]

Hey! That was the only place I had to store all of my Geocities content you insensitive clod!

What's a MySpace?

[BlueBoxSW.com]

Anyone know?

Re:What's a MySpace?

[mcgrew]

It's the area you allow between yourself and fellow hominids, and varies from location to location. In Japan, "My Space" is sparse, in rural America it's pretty large (but so are the hominids). It also varies from hominid to hominid; you need a lot of My Space for the fat stinky guy, as little as possible for the hot chick. More info here ;)

Re:What's a MySpace?

[ryantmer]

Can I nom it?

Re:What's a MySpace?

[BlueBoxSW.com]

Thank you. Much clearer now.

Re:What's a MySpace?

[Zordak]

Anyone know?

It's what Facebook was called in 2006.

Misleading title

[Darkness404]

Anyone else think from the title that MySpace has enough load (how, I don't know because who even uses it anymore) to consistently test its capacity?

16GB/second is not the same as 16 gigabits per sec

[billrp]

/. people should know better, especially anonymous readers

Reminds me of a business plan I once wrote

[Anonymous Coward]

We were going to hit our customers with enough hits to always *just* max out their account and slow their site down, making them think their site was popular and that they need to upgrade their account.

unlike your mom....

[gandhi_2]

who generates enough loa....

oh geeze. nevermind.

Re:unlike your mom....

[blai]

oh jizz

there, fixed that for you.

Re:16GB/second is not the same as 16 gigabits per

[Anonymous Coward]

Little "b" is bits. Big "B" is bytes. This is A+ material.

Re:16GB/second is not the same as 16 gigabits per

[Anonymous Coward]

To my case insensitive parser it is the same!

They did it

[OverlordQ]

by outsourcing to This Company. In additon, This Company used Stuff to do Things. After initial tests, This Company did Other Things. This Company is a leader in stuff, especially utilizing their software This Stuff. Try This Stuff Today!

Re:They did it

[Hurricane78]

Captain Grebnedlog? Is that you?

Re:They did it

[StuffMaster]

I fully approve of your message.

16 Gigabits/Second

[TheNinjaroach]

I thought that 16GB/sec seemed a little high so I checked the article. The actual network load they generated is 16 gigabits per second using 800 instances of Amazon's EC2.

Re:16 Gigabits/Second

[Anonymous Coward]

I'm sorry.. WHAT!?

16Gbit/second isn't much. It's hardly more than we handled at my previous company. A relatively small national hosting company in northern europe.

Not for testing, mind you, but 16Gbit/sec is not a lot of bandwidth. You can easily handle it with say, ~50 boxes. Considering the million of requests myspace, not even amazon, gets per second, plus all the computation they have to do in the background, it hardly seems like a .. big deal.

We're talking about 800 machines, with 3.2Kcores or so. 800 machines, each with a 100Mbit interface could easily generate 80Gbit of traffic. This test seems rather small-scale to me.

Meh.

Re:16 Gigabits/Second

[F.Ultra]

You do realise that he thought they spoke of 16GBps, that is 160Gbps.

Gotta be prepared

[Tobor the Eighth Man]

MySpace is wise to do this kind of testing and load balancing. You never know when a twelfth person might attempt to connect to the site, throwing their carefully laid plans into total chaos.

Re:Gotta be prepared

[daveime]

MySpace is wise to do this kind of testing and load balancing. You never know when a person might attempt to connect to the site, throwing their carefully laid plans into total chaos.

FTFY

A better solution - slashdot duel

[e2d2]

Pretty stupid. They could "crowdsource" by simply challenging slashdot to a duel. We'll turn your servers into smoking rubble facespace BRING IT!

Re:A better solution - slashdot duel

[jpmorgan]

It's 2010, not 1998.

Re:A better solution - slashdot duel

[Chad Birch]

Yeah, slashdot would wreck facebook and myspace, look at how much of their traffic we make up! http://siteanalytics.compete.com/facebook.com+myspace.com+slashdot.org/

We're almost off the axis, boys, keep it up!

Re:A better solution - slashdot duel

[e2d2]

Point taken. But it only takes one person to control a botnet.

Ah, so that's why

[Anonymous Coward]

There are some really great numbers in there from the performance tests, like generating 16GB/second of bandwidth and 77,000 hits/second during testing (not including the live traffic on the site at the time).

I guess all that bandwidth usage explains why the site almost always has had shitty responsiveness -- they use it up ensuring quality. That, more than the crap designs and general lameness of the site, was why I stopped using it.

Add revenue

[KevMar]

I wonder how much add revenue they generated from all those impressions?

does testing on production increase your page views?

Re:Add revenue

[eldavojohn]

does testing on production increase your page views?

"Oops, our 2009 Quarter Four revenues look a bit down ... time for some good ole fashioned load testing (ifyaknowwhatImean)."

Sure would explain how it's still hanging around.

while the article is biased, most of it is true

[mcguyver]

The article was written by an employee of the company that sold their testing software/services to MySpace. Of course they're going to have glowing reviews about their testing tools. With that said the author is right. Capacity testing in production is needed for high availability sites. Verifying your real time monitoring tools is also important.

Re:16GB/second is not the same as 16 gigabits per

[xOneca]

16 gigabits per second... 16 gigabytes per second... Doesn't matter: my server would be DoSed at only 1mb/s...

MySpace does performance testing?

[PRMan]

You would never know it based on the times I've been there...

MySpace Planning Meeting

[eldavojohn]

MySpace Engineer: Which brings us to the issue of load testing.
MySpace Exec: So do it.
MySpace Engineer: Well, we can't.
MySpace Exec: Am I missing something here? You just got done showing how stupid our users are. So just simulate them.
MySpace Engineer: Look, sir, with all due respect, we can put lol-bots up to post crap it's just that we have no way of mimicking that amount of garbage.
MySpace Exec: Well how much is it?
MySpace Engineer: Let me remind you, our previous slides showed you the magic of the MySpace machine--millions of users putting garbage in with the result being unadulterated horse shit flying out of the site. But to load test we need a lot of garbage. Several billion metric tonnes of garbage. Otherwise we just wouldn't produce the same amount of browser destroying horse shit we produce at peak loads.
MySpace Exec: Have you spoken with the City of New York?
MySpace Engineer: Sir, twenty five New York Cities wouldn't produce the amount of garbage we need.
MySpace Exec: Holy shit.
MySpace Engineer: Yes, this indeed requires a shitstack of biblical proportions.
MySpace Exec: What're we gonna do?
MySpace Engineer: Well, to solve this problem we turned to the motherload of bullshit. The one thing that everyone keeps endlessly spewing garbage about.
MySpace Exec: The Cloud!
MySpace Engineer: Bingo.

Re:MySpace Planning Meeting

[Anonymous Coward]

LOL reminds me of this

wait, wait, in addition to the live traffic??

[Anonymous Coward]

> performance tests, like generating 16GB/second of bandwidth and 77,000 hits/second during testing (not including the live traffic on the site at the time)

Ah yes. Testing massively dangerous eventualities against your live, running hardware. The old Chernobyl trick...

If you want a REAL load test

[nurb432]

Just post a story here on slashdot that there is free porn.

Re:If you want a REAL load test

[Anonymous Coward]

Maybe on digg or reddit, but slashdot?

Digital Contraception

[cosm]

Since the MySpace's crapnet is completely covered with trojans, if MySpace blows their load tests, does this mean they are open to successful backdoor penetration & injection attacks?

Re:16GB/second is not the same as 16 gigabits per

[Anonymous Coward]

1 millibit per second? That's not even a rising edge, is it?