Slashdot Mirror
When Is A MMORPG Beta Not A Beta?
Thanks to Skotos for its editorial discussing the definition and previous use of the 'Beta phase' in relation to MMO titles. The writer argues, overall, that "The problems [in Beta] arise because of the differing agendas of the parties involved in bringing the games to and through the Beta process." He then posits that, "well before the full featureset is in place, the complexity surpasses the point where internal QA processes are adequate to cope", but "on the other hand, for the purposes of getting high-quality feedback that tells you what is wrong and where to look in order to fix it, Beta sucks." So, it's suggested, the end result is that "[massively multiplayer] Betas become exercises in community management, usually long before the team is ready to make the transition from developing a game to operating one. Meanwhile, an increasingly jaded marketplace is judging the Beta against the same standards they judge games at launch, or even years past their launch." What, if anything, can be done to ameliorate or fix these problems?
And this is fine, especially for MMORPG games.
It will be solved when you get publishers to stop making milestone dates based on fiscal reports and start paying attention to the development process.
Rushing to beta is just as bad as rushing to release, but many developers are forced to do it because they are working on a timeline that has little room for internal review and repeated small-scale testing.
Counter-example, Blizzard Entertainment and World of Warcraft: they have been testing internally for months, and the game is already stable and polished. When it hits beta, they will be able to manage it, because their "beta" will be more stable than some games that are on store shelves right now. That kind of development process is hard to come, and only a very few large developers have the clout to tell the distributor to STFU until the product is done.
1. Don't think of beta testers as unpaid labor. Most beta testers are in it for a greedy reason - to get an early look at the next big thing. Spend more time in Alpha, working with PAID testers to refine gameplay and eliminate major bugs - use Beta testing as a final "shakedown", but realize that it's more usability testing than anything else.
2. Keep it small. If you want to spend less time working on community problems and more time fixing code, don't open the beta process to everyone and their brother. Get people who are going to play and report bugs - if they don't, don't invite them to participate in the next beta.
3. Develop comprehensive crash reporting, if possible. Tell the user what went wrong where, and give them the opportunity to send it to you via email or web. I can't imagine that MS got much good feedback from Windows beta testing when the only error people got was "General Protection Fault".
4. Listen to feedback - nothing is more annoying than filing a bug report during beta and seeing the same bug in the shipping product. Have at least two people touch each bug report to avoid the "rubber stamp" syndrome.
Of course, the fact of the matter is that a lot of MMORPG's use Beta testing as an opportunity to "hook em' early". They should offer a preview edition to do this, not get people who are more interested in playing games that squashing bugs disappointed with the quality of their product.
Blogging Weight Loss, Distance Education, and more at verlin.com
A key problem with beta feedback is lack of quality in favor of quantity. More incentive may or may not improve the signal-to-noise ratio. You can get thousands of people telling you there is an item loss bug or a phantom dupe bug or a client crash bug, but until you get someone who can describe in detail exactly how to make it happen, its more of a rumor than a bug report. A major cultural problem is the beta players who just want a free chance to play the game before its released. When they find bugs they'll either exploit them for their own benefit if possible, or complain vaguely about how the game is buggy and sucks if they can't exploit it. Reporting them to improve the product doesn't cross their mind. This sort of player would probably be the most motivated by rewards, but its unclear whether their bug reports would actually be of equivalent or worse quality than average.
This is more commonly known as an Alpha.
And the men who hold high places must be the ones who start
To mold a new reality... closer to the heart
Not for the final version, because then people who aren't in the beta will think they are at a disadvantage.
That's what I meant. The testing period should last a while so therefore those incentives would still mean something.
But, maybe you could keep those in the final version with some restrictions. It would be nice to give the good beta testers (the top x) their own special area within the game after it launches where those incentives could still be used. As soon as you leave, those extras would be removed. Of course, this would be heavily dependent on the resources and the experience of the development team, but I could imagine potential testers applying in droves if they could have something within the final game that nobody could get.
As far as the game companies not wanting tons of bug reports, perhaps they could have demerits for duplicate reports. A simplified, private Bugzilla-like database could be used to help out the process. That way, the testers would have to really keep track of what's going on, otherwise there could be consequences.
Until Slashdot fixes the funny modifier, use insightful or interesting. The poster knows your intentions.
Honestly, I think it depends on the individual players view as to when a MMORPG is no longer in beta - everyone is going to be looking for something different in a game. If the crafting system is fubar, then the MMOPRPG will still be in beta phase for the player who likes to craft; the PKer won't care at all (until he needs to buy phat lewt later in the game).
For me, I see a few items on my checklist until I feel I'm no longer paying to play a beta:
1. Billing system is operational: it stays up and working under a large load, and only charges me once for signing up.
2. I am able to log in first try: I get no time-outs or "could not find server" when I know that the server is up.
3. I can play for more than a few hours without being disconnected: and I'm being generous here. I don't mind being disconnected after 4 hours play because I decide to run into a full scale guild war with all my graphic settings set to high, but I shouldn't be disconnected when changing zones, or just running along an open area.
4. Quests/Missions etc can be finished: If I start a quest only to find that the NPC I need to talk to doesn't give me the item he/she should, then that's obviously a big red mark against the MMORPG. Q&A testers are there for a reason, dontyaknow...
5. Glaring gaps in gameplay are not fixed: I know over the lifetime of a MMORPG the gameplay (eg combat system, crafting) is going to be tweaked and adjusted to reflect what players are doing, or should not be doing, in the game - but if I'm playing in my first week of a MMORPG and players can't form effective groups because spells aren't anywhere near the level they should be, or a classes abilities are non-functional, then it's back to the beta drawing board for that MMORPG.
I could start getting specific now, about what I feel a MMORPG should be like upon release, but I'll just finish by saying I think it also comes down to what the MMORPG promised. If a MMORPG has promised housing - right up until release - and there is no housing, then it's still in beta. If the MMORPG has promised certain aspects to PvP or 'revolutionary' combat - and there these aspects do not exist - then it's beta, people! Why? Because I know that when these get implemeted on live servers these features that should have been in the MMORPG at release were, in fact, not there and they will also require some fixing/testing.
All this has led me to my current MMORPG of choice: Final Fantasy XI. It's been the first MMORPG in a long while where all the above criteria have been met; and then some. It was so refreshing to play a stable game at the (North American) launch.
While that does work, it's mostly on the heads of designers, developers and QA to get a product out of beta.
Anarchy Online had incentives like this through beta and still shipped in a state that left many people giving it the dubious title of one of the worst launched MMORPGs.
It should be made ABSOLUTELY CLEAR to the end users that they are TESTING THE ENVIRONMENT. Don't allow accounts for every person, their brother, sister, dog, and cat just because you want to load test your system. Load tests are easy to simulate.
In addition to having very obvious ways (like a gigantic push button on the screen that says
"REPORT BUGS BY CLICKING HERE" or some such) to report bugs, inconsistencies, and other things.. make sure that people are given credit where credit is due, for their testing work.. and send out WEEKLY surveys to ALL testers.. and they MUST be filled out and returned within a certain time frame, otherwise the user is cancelled from the beta program.
These are ways to get quality testers. Even if they have no experience in software testing, you're at least going to get valuable input. Of course, people with software testing experience would probably be preferred, but also people with experience on other MM worlds would be great, too.. with the addition of several people who haven't the slightest clue (that's the best way to find bugs.. have people who don't know what they are SUPPOSED to do to get things to work right try things...)
Make varied groups of testers travel together, presuming it's a traditional RPG... so you have people with test experience, people with MMrpg experience, and n00bs alike teaming.. sure, there's time for free for alls, but make time required to test certain things, as well.
"Champagne for my real friends - and real pain for my sham friends!" http://ericblade.postalboard.com/
Sometimes, if the beta was a good experience, I will purchase the game, as was the case with EQ and DAoC.
Betas have become not something necessarily about testing, but rather giving a preview to all of the fans who have been eagerly watching the game progress from its conception.
If a game company wants to capitalize on it, treat it as a promotional event. I'm in marketing/advertising, and from my PoV, thats exactly what these things are. For example, some games have given everybody tons of cash, lots of XP and skill points on the last day and let them go nuts. This is the kind of thing that can hook you customers. You see, if they get a taste for the high level stuff on the last day, they might be more willing to shell out the $60 for the box, and all the monthly fees necessary to get to the high level again.
Perhaps the concept of the beta-test has evolved.
Buy Steampunk Clothing Online!