Slashdot Mirror
Overcomplicated MMO Betas
Heartless writes "On the heels of Vanguard's beta 1 announcement, Heartless Gamer blog has an article looking into why MMO beta processes are overly involved and detracting from the game they are meant to improve. From the article: 'But why even have such a process in the first place? If they honestly think they are going to get any sort of actual *testing* (I use the term loosely) from an over-hyped MMORPG community... they obviously failed basic MMORPG sociology. I could link hundreds of beta leaks and broken NDA contracts, but what would be the point? What you need to know is the fact that betas are infiltrated by those that want sneak peaks at the game. Definitely not by those that truly wish to test the product. Internal testers and paid testers have proved for years to be able to produce very finished products in the single player market.'"
In beta tests the company puts in a few neat features that will attract players. They let people play it for free to stress test the servers, not look for bugs. And hopefully get some fanboys that got really excited about a few gimmicks to promote the game for them. lol, NDA even the company that put it out doesn't care that much about it. They know for a fact it will be broken. It is free marketing.
10,000 players...
1% genuinely beta test
That's a 100 person QA team - far bigger than the typical MMO will ever see.
Now up those numbers to:
50,000 players...
1% genuinely beta test
5-10% vocally bitch about every weird bug and quirk they find
Now you're looking at 500 decent QA testers and another 2,500-5,000 pain in the ass guys who're maybe worth 1% of a tester each but cumulatively do still add up.
A beta test doesn't have to have every player responsibly beta testing. Sheer numbers ensure the end effect still gets met.
Besides, by public beta, the main thing that should be getting tested is load and the weird load quirks caused by 5,000 players all deciding to try the same exploit etc. That, whether they're good testers or bad, still happens. Arguably it happens even better if they're "bad" beta testers as they're more likely to do things they "shouldn't".
In that order. You don't get many useful bug reports from the large betas (you do from the smaller stages), but you don't know how it will handle release type number of users until release, unless you beta it.
I still have more fans than freaks. WTF is wrong with you people?
perhaps you don't understand the term "Load Testing", to properly stabilise a persistant world you have to put the servers under conditions that are similar to how they would be after release. The only real way to Load test an MMO is to have actual people playing the game, sure, you could populate the world with a basic AI, but you wouldn't get the same situations that a human would get into. perhaps the Article Writer knows next to nothing about MMO development and is just pissed off because he couldn't get a Beta copy of City of Villians or something.
/. is overrun by bed-wetting elitist nerds
let it be known, for anything other than servers, a *nix OS sucks
There are a few things you will find difficult to stress test properly. It takes a full scale assault by potential users to see how well your hardware infrastucture stands up. In single player games this is a non-issue. In even multi server small scale multiplayer games it's a non issue as well. But when you have 64+ connections to a server then you have to see if you theoretical test bear out in reality.
"There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy."
The point of betas of mmorpgs is advertising. Nothing more, nothing less. It is very difficult to actually test the games, let alone have your suggestions heard in the environment set up by the game companies. They serve the same purpose as game demos released a few weeks before the release of a prominent single player game, which is to drive excitement and anticipation of the final product. I am part of a beta testing group for Activision, which stays together from game to game, and is a smaller, more intimate group. We are able to actually test and improve games (we have worked on COD, COD:UO, RTW, THUG 2, and many others published by the company), but in the environment produced by mmorpg companies this is not the goal.
In the beginning the universe was created. This made a lot of people very angry and is widely considered as a bad move.
It's like giving out free maryjo samples.....a dealer is bound to get some instant new business. Heck, they could get a commercial on every TV show for free and it wouldn't be near the value of 1000 potential customers getting a taste. And not a taste with 1 million others but a very tight availability taste.
Also, as a programmer, I can say you can unit test till you are blue in the face, but it only takes a user 5 minutes to find a bug you'd have never tested for.
If only one major bug is caught that's huge icing on the marketting and catching customers angles.
But I think there is one factor we overlooked...ego. If you'd spent years developing a new toy, you didn't do it to keep it locked up for yourself. You did it for someone to play with. I'm betting the rush of first players digging into the new toys you rolled out is huge! and heck, if it sucks, at least you only had to deal with the a small amount of laughs the first day. Ha!
Be swell
I WISH beta processes were more harrowing to get into. Without a more stringent front end, we'll just perpetuate this cycle of useless noise drowning out any real hope of relevant reporting.
From here:
There needs to be a good back end reporting system too. Forums do not cut it. They may work for a few hundred testers, of which maybe 75% of them would read the forums and 50% actually post. However, when the game starts stress testing the servers, the players will generate much more noise than actual signal on the forums. Most of this noise will be rehashing long standing bugs or incomplete features, requiring even other testers to skim posts so much they may miss something relevant.
In my opinion, all reporting should be done through ingame interface. And, the reporting should be based on the developers pushing specific agendas for specific results, at least most of the time. There is a lot to be gained by allowing your testers free run of the experience. This helps generate discussion about what is fun versus what is not. However, without specific focus on features and systems, some bugs and incompleteness can either never be reported, or doesn't get reported sufficiently enough to prioritize it being fixed prior to launch.
The problem with the article is that the author is assuming that the beta tests are something that they are not intended to be. The idea that the testing and late stage development of a multiplayer product is analogous to that a single player game shows a gross misunderstanding of the difference in the development process of multiplayer games.
MMO beta tests are not for fielding player responses, taking suggestions from the public, or even for bug reporting. The development team and internal QA does all of this far better than the public ever will. While it is nice when players do these things, there is simply too much input to be considered. Most bugs that are reported are false, and most suggestions are, well, unfeasible. To put it politely.
Now, what beta tests -are- useful for is information gathering and the exposure of balance issues and observing bugs that simply cannot be identified in a closed testing environment. MMOs are games that are designed to run unsupervised with thousands of players, and the only way to ensure proper functionality and proper balance is to open up the doors to people who will behave in a way that is consistent with the paying public. Behavior from a smaller group can be extrapolated to the larger buying public and it is this observation of the overall system that is the most useful information to developers. It is not, as the article suggests, that developers are relying on testers to do the job of internal QA.
There is also immeasurable value in data mining information on character choices, the economy of the game, what aspects of the game that people are choosing, which they are ignoring, where they go the most often, where they gather, what they're fighting, how they are playing the game in ways that are unexpected (and unsupported) by the design.
There are far too many factors in a game that is the scope of an MMO to deal with exclusively in internal testing. The data that is gathered when people simply play the game is extremely valuable and cannot be simulated or reproduced in any other way.
---
EJC
Actually some MMO players generally UNDERESTIMATE what hours they play, when and how powerful/weak their PC rigs are. You have to consider the users who stay logged on 24/7 'because they can' or because they run AFK player-run shops or whatever. If the servers stay up 24/7, theres someone out there who will stay online 24/7.
How much work is it to review countless beta applications? I have no solid numbers, but there is no way they can convince me that it doesn't take away from the game development.
Plug data into an Excel worksheet, randomly pick X number out and thats Group 1. Sort the data by 'average time spent playing games' and thats Group 2. Etc, etc, etc. Any programmer who can write a MMO game can write a program that can automate this.
The idea of NDAs is also hard for me to understand. World of Warcraft had no problem without one.
Actually, WoW didn't have a NDA because that wasn't a beta. It was free marketing. And the 'closed beta' prior to it DID have a NDA. Given how poorly the servers did at launch day, the fact that they SERIOUSLY did not take that last 'beta' seriously is clear.
WoW beta only suffered from too much interest, but Blizzard did a remarkable job of eventually getting 500,000 testers online.
Bolding by me.
Sigil will be balancing this game as any other MMORPG... over time!
Yeah other MMO games like WoW and FFXI tried before and nearly killed themselves. WoW's player run economy is a joke since anyone who can use a keyboard could craft and all but the most extreme basic materials were sold by NPCs. On top of that people were hitting the level 50 limit within weeks of the games launch and the best equipment is all obtained from time consuming, pro-hardcore instances with drop rates on par with winning the lottery. Its the exact opposite with FFXI. Leveling up in FFXI is considered to be the worst 'grind' in out of every other MMO out there, crafting is a near impossibility without learning economics 101 and accounting 101 not to mention the Chinese 'gilfarmers' screwing around with inflation. The best equipment either costs more money than your day job's wages or is dropped from an impossible to solo monster and probably has a bad drop rate.
The general rule of thumb for MMOs is that time does not cure all. Give WoW a year and people will be bitching about lack of things to do after hitting level 60 on every class and race combination. Give EQ2 a badly designed economy and one year later SOE will start acting desperately... oh oops, that already happened.
Quantity doesn't always equal quality. If that were the case, we'd still be using the old no-ranking search engines on the Internet, and Google's attempt at sorting out by relevance would have silently failed. At some point, you'd rather just get the actual info, and not scroll through 10 pages of crap before you find anything relevant. One more guy posting "my class sucks" threads is just more noise, not more signal.
In other words, when I Google for something, I'd rather have 1 link that is exactly what I want, than 100,000,000,000 irrelevant links. The same goes for beta-testing, _if_ the goal is actually to beta-test, and not just to get some free publicity: I'd rather have just 50 people actually professionally looking for bugs, than 50,000 whining about everything else.
Having 500 people who genuinely test for bugs, is _worthless_ if their signal is drowned in the noise from 50,000 people posting like there's no tomorrow about how your game sucks ass because his Priest doest't _start_ with the Mages' level 50 spell. (That's sadly not even a joke. Something Awful once had a parody of an open letter to Sony, in which they asked for really ludicrious stuff, including _literally_ that a level 1 priest should start with the most powerful mage spell. Much to their surprise, they got a helluva bunch of emails aggreeing wholeheartedly.) Or how it sucks ass and is unbalanced because it doesn't _force_ everyone else to group with his Priest that bought everything _except_ healing/buff spells. (Add a long circular-backpatting whine about how players are idiots and don't appreciate how useful that priest is with his mace alone.) Etc.
And it goes downhill from there. The guy who discovered a bug and filed it, will start _one_ post. The guys arguing that their characters should have 100% resistance to damage and an insta-kill spell that costs no mana, will start one per day. And more often than not, spill into the other topics too. (Surely a post about how a mage spell sometimes fails with no explanation, not even a "your spell was interrupted" message, is _the_ right place to post about how either (A) you mages had it too good and it was about damn time that spell got a downgrade, or (B) about how we mages are the whipping boys of the devs, and they downgraded yet another of our spells. Doom, gloom, run for the hills, and all that.)
Welcome to the wonderful world of looking for the proverbial needle in the haystack.
A polar bear is a cartesian bear after a coordinate transform.
Wow, now we are having bloggers post links to their own blogs to drive up traffic...and to make it worst the thing is hosted on blogspot. I mean seriously, at least get your own DNS and route it or buy some shared spaced.
Now onto the topic at hand, this just shows that 90% of bloggers are talking out of their ass. In this case he really misses the point of beta testing in an MMO. There are several very good reasons and they are reasons that are necessary to test. Later stages in almost all MMO beta test are load tests. They could care less at that point who reports bugs, they just want to make sure the servers don't go kaput when the game launches.
Another problem is you need a large number of people to beta test any MMO. There are literally hundreds of possibilities when a game is in the beta mode (if not thousands) for character development. Look at World of Warcraft. You have to test every race with every classes. You have to test quests at multiple levels, you need to test raids and dungeons. There is a lot more to beta in a MMO then in a single player or even regular multi-player game.
Not to mention the fact that the Beta taste gives you the chance to hook all those players and then start charging them. It also gives you the chance to get free word of mouth advertising. Open betas are a bit more of a joke on other games, but in those situations companies release "demos" to perform essentially the same task without calling it a beta. I have beta tested a regular game and an MMO. Let me say it is much bet to use a community of people in the tens or hundreds of thousands to test your game then to use your few hundred employees with an MMO. Like I said before, the sheer scale of MMOs makes fully closed Beta's an unreliable and ineffective means to test the game.
"Some days you just can't get rid of a bomb."