Exploring the Current State of Beta Testing

Karen Hertzberg writes "Since the earliest days of MMO gaming, beta testing has played a pivotal role in the success or failure of our persistent worlds. We've come a long way since the initial tests of Ultima Online and The Realm, but what role do our current beta tests play in the potential outcomes of unreleased titles? To answer this question, Ten Ton Hammer turned to current and former beta decision makers at Cryptic Studios, NetDevil, Sony Online Entertainment, Funcom, and Mythic Entertainment. Some of their answers — and the information they reveal — may surprise you."

Comments

There Really Are Some Gems in This Article

[eldavojohn]

That I wish the game companies and everyone would learn from:

[Beta] seems to be more about marketing the game and hyping it up.

So completely accurate.

Honestly I feel that nowadays the developers get confused and can not find a happy medium between listening to every single testers opinion and listening to nobody. In some beta tests I have felt the community was highly ignored while in other tests I have felt that the developers tried to cater to every single tester. Both are and were recipes for failure.

Beta tests now are glorified demos for the games. It used to be so much different. I really hate what's been done to them, and the only way to be a 'real' tester now is to get into the alpha's or in some rare cases early closed betas.

Backing up the first quote and pointing out what Betas have become.

I wish there were more people submitting bug reports, but that's the way it goes with beta, and weâ(TM)re still finding them regardless. Besides, I need all types [of players]. I need the exploits so we can find them ... I need the jerks.

At first I was shocked they would want the scripters and botmakers on so early but I soon understood that you want to catch these serious things as early as possible to fix them because:

By the time beta begins, you've made decision after decision that have compounded on each other. Your assumptions' assumptions' have assumptions about what your game is. The whole product, systems, content, operations, marketing, PR, community ramp, you name it -- is built upon them. Changing core assumptions about the product itself is unlikely to be possible without significant delays, costing progressively more money per month. (Remember, the months toward the end of the dev cycle are the most expensive ones by far.)

Ultimately, you need to believe in your product before you conduct any sort of open beta or release a demo. You can message to players all you want that the game is a "work in progress" and that many things will change before final release, but that wonâ(TM)t stop them from making judgments about the title based on their beta experiences.

I believe that's the fundamental reason behind Blizzard's horrid schedule slippages.

It is disappointing to say, but testing has become a bit of a joke, and I feel that the current crop of recent games are a reflection of that. So many games are being released incomplete (as far as hyped features go) and containing issues that should have been picked up and resolved during the closed beta phase at the latest, but this isn't happening.

It's the classic cash in while you still can mentality that has seen the release of so many unstable games only to have the servers shut off or merged down within a year.

Here's to hoping the gaming industry finds and reads this article ... we're in a bad spot right now.

Re:There Really Are Some Gems in This Article

[Chabo]

Beta tests now are glorified demos for the games. It used to be so much different. I really hate what's been done to them, and the only way to be a 'real' tester now is to get into the alpha's or in some rare cases early closed betas.

Backing up the first quote and pointing out what Betas have become.

IMO, the point of an open beta isn't to make client-side changes, because the developers can make all of those changes based on alpha or closed-beta testing, much earlier in the development process. By the time a product reaches beta stages, it's essentially done, and just needs a small amount of polish before release.

The real point of an open beta, especially for MMOs, is stress-testing the game servers. An open beta by definition tries to get everyone possible to play the game, so in addition to being a demo for the game, you're also trying to debug any bugs in the server system that simply can't be found by testing with just a few clients.

This was about all Blizzard was concentrating on with WoW during the open beta, and we can even see this outside the genre; when Valve had the pre-release demo of Left 4 Dead, they were testing what happens when thousands of gamers used their brand new matchmaking system. It had some issues that have been largely resolved by now, but they just couldn't test it that easily without that kind of hammering.

Re:There Really Are Some Gems in This Article

[Darinbob]

[Beta] seems to be more about marketing the game and hyping it up.

So completely accurate.

It has annoyed me in the past when people have said they were "lucky" to be in a beta, or grumbling that only a select few were chosen. It seems that people treat being in a beta as a perk or reward, which means that they treat the beta like an opportunity to learn the tricks before everyone else, or use it like a badge of honor. That's completely the wrong attitude, as in normal software being a beta user means a lot of hard work and attention to detail. Being in a close game beta should not be about having fun.

Re:There Really Are Some Gems in This Article

[Pollardito]

You're not going to get "a lot of hard work and attention to detail" from the testers of your commercial product unless they're being paid. The fun of playing the game early is a form of payment, but if you're asking them to forgo that fun in order to only do the work part then you're insane. There's definitely an imbalance though, with some testers who don't bother to submit bug reports at all, and also a lot of companies that don't bother to listen to the feedback they do get.

Re:There Really Are Some Gems in This Article

[DerekLyons]

You're not going to get "a lot of hard work and attention to detail" from the testers of your commercial product unless they're being paid. The fun of playing the game early is a form of payment, but if you're asking them to forgo that fun in order to only do the work part then you're insane.

I'm guessing you've never actually part of gaming beta. I all of the ones I've been in (about nine so far), finding people willing to work hard and pay attention to detail has been pretty easy.

Re:There Really Are Some Gems in This Article

[Austerity Empowers]

I don't know, I've been in closed beta for several products lately, Warhammer Online, Age of Conan and Tabula Rasa.

I submit bug reports out the wazoo (and believe me, all 3 of these have had a few bugs). I never see anything come of them, even the major ones. I try to view being in the beta as a privilege for which I owe good bug reports.

On the other hand, you are supposed to have fun. Any time I am not having fun, is just as much of a bug as when things don't work.

I've always got the feeling that the game was going to ship no matter what I found. I was just there to load the servers and stress the network so they could make the launch less hokey. I've found blatant exploits, item dupes, countless serious quest bugs, etc. I never saw any movement on any of those, nor was asked to clarify or repro. That kind of tells me a lot about how my input is being considered. It really is just a marketing thing that the developers have been able to put to small use. Beta's rarely start early enough, nor is there sufficient in-house support for beta testers for it to really be effective.

Re:There Really Are Some Gems in This Article

[fractoid]

I believe that's the fundamental reason behind Blizzard's horrid schedule slippages.

(Warning: rabid Blizzard fanboy post ahead:) In the early days, yes, Blizzard had some pretty bad schedule slippages. Unlike most companies, they correctly identified the problem to be the schedule, not the slippages. The reason that their games are still regularly played over 10 years after release is the uncompromising attitude towards quality. That's why these days they keep everything completely dark until a game's already been in development for a couple of years, and they don't commit to a release date until a couple of months before that date.

Games are art, and artwork isn't done until it's done. You can schedule an accounting app, because it has a strict function that's determined by an arbitrary set of rules. You can't schedule 'finishing' a game because the completion criteria are incredibly fuzzy and vauge. "It has to be fun." "The storyline needs to be engaging." "The player has to feel significant." "The world needs to be realistic." If you don't give this stuff enough time to be iterated over and refined, you end up with a bodgy game no matter how good your tech is.

Re:There Really Are Some Gems in This Article

[Opportunist]

It has changed, though, if you ask me.

I've been in a stretch of MMO CBs within the last decade or so. A decade ago, you had some pretty good chance to actually know a fellow tester or two. From another game, or because of the way the CB slots were distributed.

Today, CB slots are more often than not some sort of raffle prize. You have people who join the game forum (or, IMO worse, some MMO forum) where then some raffle is held with CB slots as the prize for the lucky winners.

What kind of "quality" do you expect to get out of that?

Once Beta, it's Done

[MarioMax]

At least in the case of SOE, once a game is in Beta, it's basically done. They just want marketing and bug reporting. The devs are extremely reluctant to change anything once their product is in Beta, no matter how fundamentally flawed the product is in the first place.

Halfway just for PR, Bad "beta testers"

[TibbonZero]

By the time most of these games hit Public Beta it's really just stress testing the servers a bit (but never enough) and working on some tuning stuff. Mainly I've seen this by this stage it's really just about getting a solid buzz going around the game. Most users will have broken their NDAs by this point (as we saw happen recently in WotLK), but the entire point is just to get hype going for the game.

Until recently I was working for a game-industry related company, and we had a lot of close interaction with gamers and the game companies. I'm reading the article fully right now for some more of the developer/publisher-perspective details however.

Half the problem is that most of these gamers suck at betatesting. They don't want to file bug reports, they want to play the game free/early so that their guild can get a head start on others. the number of users that I've seen rant about a game having downtime turning beta, doing server wipes, etc... They weren't complaining because they couldn't get enough bug reports in, but because they couldn't get into their Raid.

Because of pressure on various fronts, most of these games are released with insufficient server architecture, horrid bugs, and critical balance issues. This is the stuff that should be stomped out during beta, but it isn't. Beta isn't about testing, its about PR and hype. Wish it was some other way. If i was developing an MMO I'd want to disable users accounts that didn't file bug reports properly, but I know that doesn't do well for the PR side.

People feel entitled to their games, and even more entitled to a chance to play for free. As expected, its not uncommon for some big players in the game industry to give beta accounts to people who run big guilds, but don't necessarily put in bug reports.

Re:Halfway just for PR, Bad "beta testers"

[billcopc]

I think if game companies really wanted to get their value out of beta testers, they would implement simple, trivial changes like requiring a questionnaire after every N hours of game time. I'm not saying bug reports, but at least a 10-minute form where developers can ping the gamers for feedback, at least once or twice weekly.

If the kids can't take 10 minutes of their time to help improve the game they sink 20 hours a week into, they can fuck right off and wait for retail. They will buy it anyway, and if they don't, well fuck 'em. That's business.

Re:Halfway just for PR, Bad "beta testers"

[Bieeanda]

In several of the betas I've been in over the last few years, I've been faced with brief questionnaires and comment spaces at the end of every mission/quest/hallucination and quite often upon logging in or going to quit. If anything, their frequency in the early game tends to be annoyingly high, thanks to all of the quests designed to baby-step you through the interface.

But yes, most people who sign up for betas are absolutely fucking terrible at it. They have no QA training, and their posts on the official testing boards are usually 100% noise. It's even worse when they're dyed in the wool fanboys, because you'll never get useful criticism out of them.

Opposite reaction in Guild Wars

[TinBromide]

So, they talked about "good" betas and betas that opened miserably and killed a somewhat polished game, (auto assault, IIRC) that was opened too early. What about a game that the beta is BETTER than the released game?

I was part of the open beta of guild wars(i.e. i pre-ordered and was part of the PR wave of beta). It was awesome, everything was fun, it was clear that it wasn't finished, but the missions were OK, the PvE was tolerable, but the PVP was phenomenal. When it was time to release, I fired up my copy and found like 2 skills at the first skill trainer. Approximately 750 in game hours later, it was possible to recreate the PVP experience I had during beta...

750 hours in missions that are only OK and tolerable PvE that turned to miserable at the snails pace that they made you try it. Guildwars isn't a monthly thing based fee, so they gained nothing, absolutely zero, by forcing you to put 750 hours into the original campaign to get back to the fun of the open betas. By then, they had lost a very large portion of their user base and the beta users were not the the majority of the major adopters. If they had released the game we "beta tested", it probably would have been a runaway success instead of the third rate game it is today. Also, because the PVP players left for greener pastures (battlefield 2 so you can have an idea of what that crowd was), current PVP metagame is a pale imitation of what it once was and could be.

For the record, about 2 years after release, they made enough changes so that a new player can jump into that old timey fun.

One Day...

[koterica]

One Day, Beta Testing will be out of Beta.

Re:The Realm!

[Calmiche]

Sadly, it's not as dirty as it sounds.

The wrath was one of the rarest (early) swords in the game. You could enchant them with certain spells to increase strength and damage. However, the enchant spell had a chance of destroying the sword. I believe that a four enchant sword (4-banger) had a 80% chance of destroying the sword. For a time it was possible to get 5 enchants on a sword but the chance of getting 5 enchants was between 2% and 5%.

So, if you played a lot and were high level, you could find 2-3 Wraths a month on average. Then, you could burn through about 20 to 50 of them to get a 5 enchant sword.

Does anyone remember what "beta" meant?

[swordgeek]

There was a time when alpha meant "feature-complete, but broken." Beta meant "OK, let's get the last few bugs out before release."

Later, as code got more complex, beta usually went through a few phases. That was fine. Also, the beta testers were generally professionals, with some exceptions. (it's always a good plan to get real users who can break the unbreakable. Just make sure you don't count on them exclusively.)

Nowadays, what companies put through 'beta testing' is rarely alpha-code. Feature complete? Maybe, maybe not. Realistically, the second release candidate onwards through the first post-release patch should properly be considered beta, because the number of products that are even usable until after the first patch are minimal.

Ultimately, quality code doesn't pay in almost all commercial cases. Get enough untrained end-users to find the worst of the show-stopper bugs, and then release the code and start making money. Once in a long while, your product will be so bad that it falls on its face--but that's the exception, and will probably still cost less than professionally beta-testing all of your products to a high level of quality.

Tester for the last 10 years

[le_sean_moon]

I've beta tested plenty of games over the last ten years, but I didn't realize it until after I paid for them

The future isn't pretty

[Arimus]

Just thinking to the future; when we get games with direct neural interfaces or even decent force feedback systems who will want to the beta tester then?

One wrong function call and the word 'headache' could gain a whole new meaning...