Slashdot Mirror
Exploring the Current State of Beta Testing
Karen Hertzberg writes "Since the earliest days of MMO gaming, beta testing has played a pivotal role in the success or failure of our persistent worlds. We've come a long way since the initial tests of Ultima Online and The Realm, but what role do our current beta tests play in the potential outcomes of unreleased titles? To answer this question, Ten Ton Hammer turned to current and former beta decision makers at Cryptic Studios, NetDevil, Sony Online Entertainment, Funcom, and Mythic Entertainment. Some of their answers — and the information they reveal — may surprise you."
You're not going to get "a lot of hard work and attention to detail" from the testers of your commercial product unless they're being paid. The fun of playing the game early is a form of payment, but if you're asking them to forgo that fun in order to only do the work part then you're insane. There's definitely an imbalance though, with some testers who don't bother to submit bug reports at all, and also a lot of companies that don't bother to listen to the feedback they do get.
So, they talked about "good" betas and betas that opened miserably and killed a somewhat polished game, (auto assault, IIRC) that was opened too early. What about a game that the beta is BETTER than the released game?
I was part of the open beta of guild wars(i.e. i pre-ordered and was part of the PR wave of beta). It was awesome, everything was fun, it was clear that it wasn't finished, but the missions were OK, the PvE was tolerable, but the PVP was phenomenal. When it was time to release, I fired up my copy and found like 2 skills at the first skill trainer. Approximately 750 in game hours later, it was possible to recreate the PVP experience I had during beta...
750 hours in missions that are only OK and tolerable PvE that turned to miserable at the snails pace that they made you try it. Guildwars isn't a monthly thing based fee, so they gained nothing, absolutely zero, by forcing you to put 750 hours into the original campaign to get back to the fun of the open betas. By then, they had lost a very large portion of their user base and the beta users were not the the majority of the major adopters. If they had released the game we "beta tested", it probably would have been a runaway success instead of the third rate game it is today. Also, because the PVP players left for greener pastures (battlefield 2 so you can have an idea of what that crowd was), current PVP metagame is a pale imitation of what it once was and could be.
For the record, about 2 years after release, they made enough changes so that a new player can jump into that old timey fun.
Is it sad that I am more likely to recognize you and your posts by your sig than your name or UID?
I don't know, I've been in closed beta for several products lately, Warhammer Online, Age of Conan and Tabula Rasa.
I submit bug reports out the wazoo (and believe me, all 3 of these have had a few bugs). I never see anything come of them, even the major ones. I try to view being in the beta as a privilege for which I owe good bug reports.
On the other hand, you are supposed to have fun. Any time I am not having fun, is just as much of a bug as when things don't work.
I've always got the feeling that the game was going to ship no matter what I found. I was just there to load the servers and stress the network so they could make the launch less hokey. I've found blatant exploits, item dupes, countless serious quest bugs, etc. I never saw any movement on any of those, nor was asked to clarify or repro. That kind of tells me a lot about how my input is being considered. It really is just a marketing thing that the developers have been able to put to small use. Beta's rarely start early enough, nor is there sufficient in-house support for beta testers for it to really be effective.