When Is A MMORPG Beta Not A Beta?

Thanks to Skotos for its editorial discussing the definition and previous use of the 'Beta phase' in relation to MMO titles. The writer argues, overall, that "The problems [in Beta] arise because of the differing agendas of the parties involved in bringing the games to and through the Beta process." He then posits that, "well before the full featureset is in place, the complexity surpasses the point where internal QA processes are adequate to cope", but "on the other hand, for the purposes of getting high-quality feedback that tells you what is wrong and where to look in order to fix it, Beta sucks." So, it's suggested, the end result is that "[massively multiplayer] Betas become exercises in community management, usually long before the team is ready to make the transition from developing a game to operating one. Meanwhile, an increasingly jaded marketplace is judging the Beta against the same standards they judge games at launch, or even years past their launch." What, if anything, can be done to ameliorate or fix these problems?

Comments

When is it not a beta?

[AtariAmarok]

You know it's not a beta any more when you stop finding "SYNTAX ERROR: LINE 6778 INVALID CHARACTER AFTER ;" in treasure caves.

How About Incentives?

[illuminata]

When a player successfully finds and reports a problem in the game, they could be awarded with extra stats, items, in-game money, etc. In fact, maybe real prizes like exclusive clothing, figurines, developer autographs, cash credit at an affiliate site, or just plain old cash would help out too.

The prizes probably should be rated by severity of the issue that they brought to light. As they find more problems, they would get more prizes.

It probably wouldn't hurt to give a special thanks page, possibly even a rankings page so that it could be turned into a true competition. The testers could receive a special title for when the game is officially released, too.

Gamers tend to like free stuff that nobody else can get and respect within the game that they're playing. Any one of these ideas should be able to feed both needs.

Re:How About Incentives?

[1019]

Blizzard does this in a sense with their "Blizzard Friends" program where if you are selected for a Beta and you consistantly provide original and reproducable bugs in the game, you get added to the Friends list. The users on this list are automatically entered into the next Blizzard beta (whatever it will be..in this case the WoW Beta) but then they have to re-earn the right to be on the list for that one and so on and so on.

I don't know if rewarding bug-hunters with stat/item upgrades would be in the best interest of trying to develop an environment so dependant on balance, but I think Blizzard has a good idea.

I was selected for the Warcraft III: Frozen Throne Beta but unfortunately did not have the time to be a persistant tester, despite it being very fun.

Here's hoping for WoW!

Possible Solution

[Josiwe]

A company like Square-Enix could very easily maintain a stable of beta testers across 5-10 games, rotating in fresh blood every time to keep the pool alive as certain testers grow older and have less time to beta test. After each beta cycle non-responsive or poor quality beta testers could be thanked and excused from service - they could even maintain two pools, one for actual testing and one for winnowing out efficient beta testers.

Think of it like a job interview where the compensation is getting to play cutting edge games way ahead of the public as well as getting the developer's ear. And the company gets a small pool of very good testers (maybe 300) that they can count on to do good work and still be around for FFDQXXVI Online.

Re:Possible Solution

[MachDelta]

Atari has already started something like that. Its over at Betatests.net. The basic idea is that you register an account, and then apply to various games to undergo beta testing. As you report bugs (though BetaTests.net) and generally be a good little tester, you gain 'points' and go up in 'rank'. The more points you have, and the higher rank you get, the more likely you are to be chosen for other tests by the developers surfing through your profile.

Unfortunatly, its off to a bit of a rocky start. For one, there are a grand total of three things to test. And one isn't even a friggin game! There was a fourth game, Unreal 2: XMP (eXpanded MultiPlayer), but that test is now closed. The game/free addon was released publicly, and the existing crew of testers are being kept onboard to test future patches. Speaking of which, we're (yeah, i'm one) in the middle of testing a patch right now.
In addition to the severe lack of games, the system doesn't even work properly. Bugs submitted through the website are generally ignored, or horribly delayed as they shuffle from Tester -> Publisher -> Developer and back. During the U2XMP test, we found it was *MUCH* faster, easier, and simpler to just use the forums to directly contact the devs. (And speaking of forums, BetaTests.net is supposed to provide forums for each test, but those were broken at the time of the XMP test too. So we wound up using a well respected fansite's forums instead). And to top it all off, due to the horribly inefficient reporting scheme, none of the developers seem to be taking (more like wasting) the time to actually give people 'points' to move up in rank.

So all in all... good concept, but excecution is decidedly more difficult. Hopefully Atari can get things turned around in short order, but the system just isn't sustainable in its current state. Maybe once Atari figures out how to plow through this mess, other publishing groups will start similar projects. And who knows, maybe someone will eventually propose a universal database of testers?

Anyways... just FYI, it does exist, albet tenuously

How to solve it?

[Edgewize]

It will be solved when you get publishers to stop making milestone dates based on fiscal reports and start paying attention to the development process.

Rushing to beta is just as bad as rushing to release, but many developers are forced to do it because they are working on a timeline that has little room for internal review and repeated small-scale testing.

Counter-example, Blizzard Entertainment and World of Warcraft: they have been testing internally for months, and the game is already stable and polished. When it hits beta, they will be able to manage it, because their "beta" will be more stable than some games that are on store shelves right now. That kind of development process is hard to come, and only a very few large developers have the clout to tell the distributor to STFU until the product is done.

Re:the real question

[Lowtekium]

The real question most MMORPG gamers would ask is "When is a MMORPG not in beta?"

That's an easy one. To quote a developer of one of the currently largest MMOGs I met last year, "When the money runs out, its time to launch!"

things that could be done

[XO]

It should be made ABSOLUTELY CLEAR to the end users that they are TESTING THE ENVIRONMENT. Don't allow accounts for every person, their brother, sister, dog, and cat just because you want to load test your system. Load tests are easy to simulate.

In addition to having very obvious ways (like a gigantic push button on the screen that says
"REPORT BUGS BY CLICKING HERE" or some such) to report bugs, inconsistencies, and other things.. make sure that people are given credit where credit is due, for their testing work.. and send out WEEKLY surveys to ALL testers.. and they MUST be filled out and returned within a certain time frame, otherwise the user is cancelled from the beta program.

These are ways to get quality testers. Even if they have no experience in software testing, you're at least going to get valuable input. Of course, people with software testing experience would probably be preferred, but also people with experience on other MM worlds would be great, too.. with the addition of several people who haven't the slightest clue (that's the best way to find bugs.. have people who don't know what they are SUPPOSED to do to get things to work right try things...)

Make varied groups of testers travel together, presuming it's a traditional RPG... so you have people with test experience, people with MMrpg experience, and n00bs alike teaming.. sure, there's time for free for alls, but make time required to test certain things, as well.