Slashdot Mirror
Overcomplicated MMO Betas
Heartless writes "On the heels of Vanguard's beta 1 announcement, Heartless Gamer blog has an article looking into why MMO beta processes are overly involved and detracting from the game they are meant to improve. From the article: 'But why even have such a process in the first place? If they honestly think they are going to get any sort of actual *testing* (I use the term loosely) from an over-hyped MMORPG community... they obviously failed basic MMORPG sociology. I could link hundreds of beta leaks and broken NDA contracts, but what would be the point? What you need to know is the fact that betas are infiltrated by those that want sneak peaks at the game. Definitely not by those that truly wish to test the product. Internal testers and paid testers have proved for years to be able to produce very finished products in the single player market.'"
In beta tests the company puts in a few neat features that will attract players. They let people play it for free to stress test the servers, not look for bugs. And hopefully get some fanboys that got really excited about a few gimmicks to promote the game for them. lol, NDA even the company that put it out doesn't care that much about it. They know for a fact it will be broken. It is free marketing.
10,000 players...
1% genuinely beta test
That's a 100 person QA team - far bigger than the typical MMO will ever see.
Now up those numbers to:
50,000 players...
1% genuinely beta test
5-10% vocally bitch about every weird bug and quirk they find
Now you're looking at 500 decent QA testers and another 2,500-5,000 pain in the ass guys who're maybe worth 1% of a tester each but cumulatively do still add up.
A beta test doesn't have to have every player responsibly beta testing. Sheer numbers ensure the end effect still gets met.
Besides, by public beta, the main thing that should be getting tested is load and the weird load quirks caused by 5,000 players all deciding to try the same exploit etc. That, whether they're good testers or bad, still happens. Arguably it happens even better if they're "bad" beta testers as they're more likely to do things they "shouldn't".
perhaps you don't understand the term "Load Testing", to properly stabilise a persistant world you have to put the servers under conditions that are similar to how they would be after release. The only real way to Load test an MMO is to have actual people playing the game, sure, you could populate the world with a basic AI, but you wouldn't get the same situations that a human would get into. perhaps the Article Writer knows next to nothing about MMO development and is just pissed off because he couldn't get a Beta copy of City of Villians or something.
/. is overrun by bed-wetting elitist nerds
let it be known, for anything other than servers, a *nix OS sucks
There are a few things you will find difficult to stress test properly. It takes a full scale assault by potential users to see how well your hardware infrastucture stands up. In single player games this is a non-issue. In even multi server small scale multiplayer games it's a non issue as well. But when you have 64+ connections to a server then you have to see if you theoretical test bear out in reality.
"There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy."
The point of betas of mmorpgs is advertising. Nothing more, nothing less. It is very difficult to actually test the games, let alone have your suggestions heard in the environment set up by the game companies. They serve the same purpose as game demos released a few weeks before the release of a prominent single player game, which is to drive excitement and anticipation of the final product. I am part of a beta testing group for Activision, which stays together from game to game, and is a smaller, more intimate group. We are able to actually test and improve games (we have worked on COD, COD:UO, RTW, THUG 2, and many others published by the company), but in the environment produced by mmorpg companies this is not the goal.
In the beginning the universe was created. This made a lot of people very angry and is widely considered as a bad move.
I WISH beta processes were more harrowing to get into. Without a more stringent front end, we'll just perpetuate this cycle of useless noise drowning out any real hope of relevant reporting.
From here:
There needs to be a good back end reporting system too. Forums do not cut it. They may work for a few hundred testers, of which maybe 75% of them would read the forums and 50% actually post. However, when the game starts stress testing the servers, the players will generate much more noise than actual signal on the forums. Most of this noise will be rehashing long standing bugs or incomplete features, requiring even other testers to skim posts so much they may miss something relevant.
In my opinion, all reporting should be done through ingame interface. And, the reporting should be based on the developers pushing specific agendas for specific results, at least most of the time. There is a lot to be gained by allowing your testers free run of the experience. This helps generate discussion about what is fun versus what is not. However, without specific focus on features and systems, some bugs and incompleteness can either never be reported, or doesn't get reported sufficiently enough to prioritize it being fixed prior to launch.
The problem with the article is that the author is assuming that the beta tests are something that they are not intended to be. The idea that the testing and late stage development of a multiplayer product is analogous to that a single player game shows a gross misunderstanding of the difference in the development process of multiplayer games.
MMO beta tests are not for fielding player responses, taking suggestions from the public, or even for bug reporting. The development team and internal QA does all of this far better than the public ever will. While it is nice when players do these things, there is simply too much input to be considered. Most bugs that are reported are false, and most suggestions are, well, unfeasible. To put it politely.
Now, what beta tests -are- useful for is information gathering and the exposure of balance issues and observing bugs that simply cannot be identified in a closed testing environment. MMOs are games that are designed to run unsupervised with thousands of players, and the only way to ensure proper functionality and proper balance is to open up the doors to people who will behave in a way that is consistent with the paying public. Behavior from a smaller group can be extrapolated to the larger buying public and it is this observation of the overall system that is the most useful information to developers. It is not, as the article suggests, that developers are relying on testers to do the job of internal QA.
There is also immeasurable value in data mining information on character choices, the economy of the game, what aspects of the game that people are choosing, which they are ignoring, where they go the most often, where they gather, what they're fighting, how they are playing the game in ways that are unexpected (and unsupported) by the design.
There are far too many factors in a game that is the scope of an MMO to deal with exclusively in internal testing. The data that is gathered when people simply play the game is extremely valuable and cannot be simulated or reproduced in any other way.
---
EJC
Quantity doesn't always equal quality. If that were the case, we'd still be using the old no-ranking search engines on the Internet, and Google's attempt at sorting out by relevance would have silently failed. At some point, you'd rather just get the actual info, and not scroll through 10 pages of crap before you find anything relevant. One more guy posting "my class sucks" threads is just more noise, not more signal.
In other words, when I Google for something, I'd rather have 1 link that is exactly what I want, than 100,000,000,000 irrelevant links. The same goes for beta-testing, _if_ the goal is actually to beta-test, and not just to get some free publicity: I'd rather have just 50 people actually professionally looking for bugs, than 50,000 whining about everything else.
Having 500 people who genuinely test for bugs, is _worthless_ if their signal is drowned in the noise from 50,000 people posting like there's no tomorrow about how your game sucks ass because his Priest doest't _start_ with the Mages' level 50 spell. (That's sadly not even a joke. Something Awful once had a parody of an open letter to Sony, in which they asked for really ludicrious stuff, including _literally_ that a level 1 priest should start with the most powerful mage spell. Much to their surprise, they got a helluva bunch of emails aggreeing wholeheartedly.) Or how it sucks ass and is unbalanced because it doesn't _force_ everyone else to group with his Priest that bought everything _except_ healing/buff spells. (Add a long circular-backpatting whine about how players are idiots and don't appreciate how useful that priest is with his mace alone.) Etc.
And it goes downhill from there. The guy who discovered a bug and filed it, will start _one_ post. The guys arguing that their characters should have 100% resistance to damage and an insta-kill spell that costs no mana, will start one per day. And more often than not, spill into the other topics too. (Surely a post about how a mage spell sometimes fails with no explanation, not even a "your spell was interrupted" message, is _the_ right place to post about how either (A) you mages had it too good and it was about damn time that spell got a downgrade, or (B) about how we mages are the whipping boys of the devs, and they downgraded yet another of our spells. Doom, gloom, run for the hills, and all that.)
Welcome to the wonderful world of looking for the proverbial needle in the haystack.
A polar bear is a cartesian bear after a coordinate transform.