Overcomplicated MMO Betas

Heartless writes "On the heels of Vanguard's beta 1 announcement, Heartless Gamer blog has an article looking into why MMO beta processes are overly involved and detracting from the game they are meant to improve. From the article: 'But why even have such a process in the first place? If they honestly think they are going to get any sort of actual *testing* (I use the term loosely) from an over-hyped MMORPG community... they obviously failed basic MMORPG sociology. I could link hundreds of beta leaks and broken NDA contracts, but what would be the point? What you need to know is the fact that betas are infiltrated by those that want sneak peaks at the game. Definitely not by those that truly wish to test the product. Internal testers and paid testers have proved for years to be able to produce very finished products in the single player market.'"

Comments

Gimmicks

[danikar]

In beta tests the company puts in a few neat features that will attract players. They let people play it for free to stress test the servers, not look for bugs. And hopefully get some fanboys that got really excited about a few gimmicks to promote the game for them. lol, NDA even the company that put it out doesn't care that much about it. They know for a fact it will be broken. It is free marketing.

Weight Of Numbers

[Anonymous Coward]

10,000 players...
1% genuinely beta test
That's a 100 person QA team - far bigger than the typical MMO will ever see.

Now up those numbers to:
50,000 players...
1% genuinely beta test
5-10% vocally bitch about every weird bug and quirk they find
Now you're looking at 500 decent QA testers and another 2,500-5,000 pain in the ass guys who're maybe worth 1% of a tester each but cumulatively do still add up.

A beta test doesn't have to have every player responsibly beta testing. Sheer numbers ensure the end effect still gets met.

Besides, by public beta, the main thing that should be getting tested is load and the weird load quirks caused by 5,000 players all deciding to try the same exploit etc. That, whether they're good testers or bad, still happens. Arguably it happens even better if they're "bad" beta testers as they're more likely to do things they "shouldn't".

Re:Weight Of Numbers

[dhakbar]

As someone who has worked QA on a variety of SOE games and have had to read through player-submitted bug reports, there are FAR fewer than 1% of the player base submitting bugs in the first place, and the ones who bother cannot be considered "decent QA testers." Someone else mentioned it earlier... the beta phase is merely to hype the game. It's incredibly rare that players, even those with legitimate bugs, submit reports that are useful for QA purposes.

Marketing and stress test

[AuMatar]

In that order. You don't get many useful bug reports from the large betas (you do from the smaller stages), but you don't know how it will handle release type number of users until release, unless you beta it.

Public Beta testers aren't bug-hunters

[incubusnb]

perhaps you don't understand the term "Load Testing", to properly stabilise a persistant world you have to put the servers under conditions that are similar to how they would be after release. The only real way to Load test an MMO is to have actual people playing the game, sure, you could populate the world with a basic AI, but you wouldn't get the same situations that a human would get into. perhaps the Article Writer knows next to nothing about MMO development and is just pissed off because he couldn't get a Beta copy of City of Villians or something.

That's not the point

[wyldeone]

The point of betas of mmorpgs is advertising. Nothing more, nothing less. It is very difficult to actually test the games, let alone have your suggestions heard in the environment set up by the game companies. They serve the same purpose as game demos released a few weeks before the release of a prominent single player game, which is to drive excitement and anticipation of the final product. I am part of a beta testing group for Activision, which stays together from game to game, and is a smaller, more intimate group. We are able to actually test and improve games (we have worked on COD, COD:UO, RTW, THUG 2, and many others published by the company), but in the environment produced by mmorpg companies this is not the goal.

More than free advertising...

[Tojosan]

It's like giving out free maryjo samples.....a dealer is bound to get some instant new business. Heck, they could get a commercial on every TV show for free and it wouldn't be near the value of 1000 potential customers getting a taste. And not a taste with 1 million others but a very tight availability taste.

Also, as a programmer, I can say you can unit test till you are blue in the face, but it only takes a user 5 minutes to find a bug you'd have never tested for.
If only one major bug is caught that's huge icing on the marketting and catching customers angles.

But I think there is one factor we overlooked...ego. If you'd spent years developing a new toy, you didn't do it to keep it locked up for yourself. You did it for someone to play with. I'm betting the rush of first players digging into the new toys you rolled out is huge! and heck, if it sucks, at least you only had to deal with the a small amount of laughs the first day. Ha!

Be swell

Noise To Signal Ratio

[Moraelin]

Quantity doesn't always equal quality. If that were the case, we'd still be using the old no-ranking search engines on the Internet, and Google's attempt at sorting out by relevance would have silently failed. At some point, you'd rather just get the actual info, and not scroll through 10 pages of crap before you find anything relevant. One more guy posting "my class sucks" threads is just more noise, not more signal.

In other words, when I Google for something, I'd rather have 1 link that is exactly what I want, than 100,000,000,000 irrelevant links. The same goes for beta-testing, _if_ the goal is actually to beta-test, and not just to get some free publicity: I'd rather have just 50 people actually professionally looking for bugs, than 50,000 whining about everything else.

Having 500 people who genuinely test for bugs, is _worthless_ if their signal is drowned in the noise from 50,000 people posting like there's no tomorrow about how your game sucks ass because his Priest doest't _start_ with the Mages' level 50 spell. (That's sadly not even a joke. Something Awful once had a parody of an open letter to Sony, in which they asked for really ludicrious stuff, including _literally_ that a level 1 priest should start with the most powerful mage spell. Much to their surprise, they got a helluva bunch of emails aggreeing wholeheartedly.) Or how it sucks ass and is unbalanced because it doesn't _force_ everyone else to group with his Priest that bought everything _except_ healing/buff spells. (Add a long circular-backpatting whine about how players are idiots and don't appreciate how useful that priest is with his mace alone.) Etc.

And it goes downhill from there. The guy who discovered a bug and filed it, will start _one_ post. The guys arguing that their characters should have 100% resistance to damage and an insta-kill spell that costs no mana, will start one per day. And more often than not, spill into the other topics too. (Surely a post about how a mage spell sometimes fails with no explanation, not even a "your spell was interrupted" message, is _the_ right place to post about how either (A) you mages had it too good and it was about damn time that spell got a downgrade, or (B) about how we mages are the whipping boys of the devs, and they downgraded yet another of our spells. Doom, gloom, run for the hills, and all that.)

Welcome to the wonderful world of looking for the proverbial needle in the haystack.