Slashdot Mirror
Overcomplicated MMO Betas
Heartless writes "On the heels of Vanguard's beta 1 announcement, Heartless Gamer blog has an article looking into why MMO beta processes are overly involved and detracting from the game they are meant to improve. From the article: 'But why even have such a process in the first place? If they honestly think they are going to get any sort of actual *testing* (I use the term loosely) from an over-hyped MMORPG community... they obviously failed basic MMORPG sociology. I could link hundreds of beta leaks and broken NDA contracts, but what would be the point? What you need to know is the fact that betas are infiltrated by those that want sneak peaks at the game. Definitely not by those that truly wish to test the product. Internal testers and paid testers have proved for years to be able to produce very finished products in the single player market.'"
In beta tests the company puts in a few neat features that will attract players. They let people play it for free to stress test the servers, not look for bugs. And hopefully get some fanboys that got really excited about a few gimmicks to promote the game for them. lol, NDA even the company that put it out doesn't care that much about it. They know for a fact it will be broken. It is free marketing.
10,000 players...
1% genuinely beta test
That's a 100 person QA team - far bigger than the typical MMO will ever see.
Now up those numbers to:
50,000 players...
1% genuinely beta test
5-10% vocally bitch about every weird bug and quirk they find
Now you're looking at 500 decent QA testers and another 2,500-5,000 pain in the ass guys who're maybe worth 1% of a tester each but cumulatively do still add up.
A beta test doesn't have to have every player responsibly beta testing. Sheer numbers ensure the end effect still gets met.
Besides, by public beta, the main thing that should be getting tested is load and the weird load quirks caused by 5,000 players all deciding to try the same exploit etc. That, whether they're good testers or bad, still happens. Arguably it happens even better if they're "bad" beta testers as they're more likely to do things they "shouldn't".
In that order. You don't get many useful bug reports from the large betas (you do from the smaller stages), but you don't know how it will handle release type number of users until release, unless you beta it.
I still have more fans than freaks. WTF is wrong with you people?
perhaps you don't understand the term "Load Testing", to properly stabilise a persistant world you have to put the servers under conditions that are similar to how they would be after release. The only real way to Load test an MMO is to have actual people playing the game, sure, you could populate the world with a basic AI, but you wouldn't get the same situations that a human would get into. perhaps the Article Writer knows next to nothing about MMO development and is just pissed off because he couldn't get a Beta copy of City of Villians or something.
/. is overrun by bed-wetting elitist nerds
let it be known, for anything other than servers, a *nix OS sucks
There are a few things you will find difficult to stress test properly. It takes a full scale assault by potential users to see how well your hardware infrastucture stands up. In single player games this is a non-issue. In even multi server small scale multiplayer games it's a non issue as well. But when you have 64+ connections to a server then you have to see if you theoretical test bear out in reality.
"There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy."
The point of betas of mmorpgs is advertising. Nothing more, nothing less. It is very difficult to actually test the games, let alone have your suggestions heard in the environment set up by the game companies. They serve the same purpose as game demos released a few weeks before the release of a prominent single player game, which is to drive excitement and anticipation of the final product. I am part of a beta testing group for Activision, which stays together from game to game, and is a smaller, more intimate group. We are able to actually test and improve games (we have worked on COD, COD:UO, RTW, THUG 2, and many others published by the company), but in the environment produced by mmorpg companies this is not the goal.
In the beginning the universe was created. This made a lot of people very angry and is widely considered as a bad move.
Wow showed that load testing is pretty damn important. I douby a single user can really simulate all the factors that go into something like that.
The war with islam is a war on the beast
The war on terror is a war for peace
One thing that, while rare, is still quite aggravating is when a studio alpha tests with next to no features and then charges for the beta. I still can't find out how this does anything but alienate users, but somehow people manage to pull it off.
Perfecting Discordia
www.stevenvansickle.com
So a MMO developer should hire 20 000 ppl to test their games eh? You're a genius mate. The industry needs more people like you to sort it all out.
It's like giving out free maryjo samples.....a dealer is bound to get some instant new business. Heck, they could get a commercial on every TV show for free and it wouldn't be near the value of 1000 potential customers getting a taste. And not a taste with 1 million others but a very tight availability taste.
Also, as a programmer, I can say you can unit test till you are blue in the face, but it only takes a user 5 minutes to find a bug you'd have never tested for.
If only one major bug is caught that's huge icing on the marketting and catching customers angles.
But I think there is one factor we overlooked...ego. If you'd spent years developing a new toy, you didn't do it to keep it locked up for yourself. You did it for someone to play with. I'm betting the rush of first players digging into the new toys you rolled out is huge! and heck, if it sucks, at least you only had to deal with the a small amount of laughs the first day. Ha!
Be swell
I WISH beta processes were more harrowing to get into. Without a more stringent front end, we'll just perpetuate this cycle of useless noise drowning out any real hope of relevant reporting.
From here:
There needs to be a good back end reporting system too. Forums do not cut it. They may work for a few hundred testers, of which maybe 75% of them would read the forums and 50% actually post. However, when the game starts stress testing the servers, the players will generate much more noise than actual signal on the forums. Most of this noise will be rehashing long standing bugs or incomplete features, requiring even other testers to skim posts so much they may miss something relevant.
In my opinion, all reporting should be done through ingame interface. And, the reporting should be based on the developers pushing specific agendas for specific results, at least most of the time. There is a lot to be gained by allowing your testers free run of the experience. This helps generate discussion about what is fun versus what is not. However, without specific focus on features and systems, some bugs and incompleteness can either never be reported, or doesn't get reported sufficiently enough to prioritize it being fixed prior to launch.
The problem with the article is that the author is assuming that the beta tests are something that they are not intended to be. The idea that the testing and late stage development of a multiplayer product is analogous to that a single player game shows a gross misunderstanding of the difference in the development process of multiplayer games.
MMO beta tests are not for fielding player responses, taking suggestions from the public, or even for bug reporting. The development team and internal QA does all of this far better than the public ever will. While it is nice when players do these things, there is simply too much input to be considered. Most bugs that are reported are false, and most suggestions are, well, unfeasible. To put it politely.
Now, what beta tests -are- useful for is information gathering and the exposure of balance issues and observing bugs that simply cannot be identified in a closed testing environment. MMOs are games that are designed to run unsupervised with thousands of players, and the only way to ensure proper functionality and proper balance is to open up the doors to people who will behave in a way that is consistent with the paying public. Behavior from a smaller group can be extrapolated to the larger buying public and it is this observation of the overall system that is the most useful information to developers. It is not, as the article suggests, that developers are relying on testers to do the job of internal QA.
There is also immeasurable value in data mining information on character choices, the economy of the game, what aspects of the game that people are choosing, which they are ignoring, where they go the most often, where they gather, what they're fighting, how they are playing the game in ways that are unexpected (and unsupported) by the design.
There are far too many factors in a game that is the scope of an MMO to deal with exclusively in internal testing. The data that is gathered when people simply play the game is extremely valuable and cannot be simulated or reproduced in any other way.
---
EJC
Actually some MMO players generally UNDERESTIMATE what hours they play, when and how powerful/weak their PC rigs are. You have to consider the users who stay logged on 24/7 'because they can' or because they run AFK player-run shops or whatever. If the servers stay up 24/7, theres someone out there who will stay online 24/7.
How much work is it to review countless beta applications? I have no solid numbers, but there is no way they can convince me that it doesn't take away from the game development.
Plug data into an Excel worksheet, randomly pick X number out and thats Group 1. Sort the data by 'average time spent playing games' and thats Group 2. Etc, etc, etc. Any programmer who can write a MMO game can write a program that can automate this.
The idea of NDAs is also hard for me to understand. World of Warcraft had no problem without one.
Actually, WoW didn't have a NDA because that wasn't a beta. It was free marketing. And the 'closed beta' prior to it DID have a NDA. Given how poorly the servers did at launch day, the fact that they SERIOUSLY did not take that last 'beta' seriously is clear.
WoW beta only suffered from too much interest, but Blizzard did a remarkable job of eventually getting 500,000 testers online.
Bolding by me.
Sigil will be balancing this game as any other MMORPG... over time!
Yeah other MMO games like WoW and FFXI tried before and nearly killed themselves. WoW's player run economy is a joke since anyone who can use a keyboard could craft and all but the most extreme basic materials were sold by NPCs. On top of that people were hitting the level 50 limit within weeks of the games launch and the best equipment is all obtained from time consuming, pro-hardcore instances with drop rates on par with winning the lottery. Its the exact opposite with FFXI. Leveling up in FFXI is considered to be the worst 'grind' in out of every other MMO out there, crafting is a near impossibility without learning economics 101 and accounting 101 not to mention the Chinese 'gilfarmers' screwing around with inflation. The best equipment either costs more money than your day job's wages or is dropped from an impossible to solo monster and probably has a bad drop rate.
The general rule of thumb for MMOs is that time does not cure all. Give WoW a year and people will be bitching about lack of things to do after hitting level 60 on every class and race combination. Give EQ2 a badly designed economy and one year later SOE will start acting desperately... oh oops, that already happened.
Sign up on www.betaguild.com . I know it hasn't been updated for a while, but thats because my host is undergoing renovation and I can't change stuff. I can get you a part of the team though. I'm looking for a crack team of beta testers. In the long run, I'll figure out who's the best beta testers around, and be able to solicit them to companies. For right now, we're just in the slow growth stage for a few years.
God spoke to me.
I think the actual beta test application is one of the best applications I've ever seen really. It gave plenty of opportunity to find out what type of player was applying to the beta. I've seen Brad post on the official forums and also on community boards he does not control. I've been very impressed with how much he cares about the game.
I would agree that there are many people out there that just want to play the game and not actully test or provide feedback on the game. However I've been in many early alpha/beta tests and there are a lot of very dedicated testers that provide amazing feedback. Not a lot of people get the chance to see some of the great discussions that go on between the testers and devs in these early tests. Once a test goes to a big stress test stage the devs kind of disapear from the scene because there is a lot of noise and it becomes hard to have some good discussions. I think early testing should absolutely be a hard processes in order to find out who the dedicated testers will be. If someone doesn't like to put forth the effort to fill out some information on an application I doubt they will be willing to take the time to post feedback/bug repots on the game.
Games I've tested
Meridian 59 by 3do (Beta) - 1996
Meridian 59 (Private dev server for testing potential patches) - 1997
Magestorm by engage games (beta 1997)
Diablo (Beta and stress) - 1998
Everquest (Beta 2,3,4 and stress) -1999
Asherons Call (Beta) - 1999
Diablo II (stress test) - 2001
Anarchy Online (stress test) - 2001
Dark Age of Camelot (Beta 3,4) - 2001
Earth and Beyond Closed Beta 2001
Meridian59 Re-release 2002
Final Fantasy XI Japanese Version Private test - Feb 2003
Final Fantasy US Open Beta Test - July 2003
World of Warcraft (Alpha and Beta) 2003
Everquest 2 (Late beta) 2004
Guild Wars (Private Alpha) 2004
Matrix Online - 2004
Auto Assult Online 2005
Guild Wars (Post release private Dev servers) 2005
Other betas that can't be mentioned due to NDAs
Quantity doesn't always equal quality. If that were the case, we'd still be using the old no-ranking search engines on the Internet, and Google's attempt at sorting out by relevance would have silently failed. At some point, you'd rather just get the actual info, and not scroll through 10 pages of crap before you find anything relevant. One more guy posting "my class sucks" threads is just more noise, not more signal.
In other words, when I Google for something, I'd rather have 1 link that is exactly what I want, than 100,000,000,000 irrelevant links. The same goes for beta-testing, _if_ the goal is actually to beta-test, and not just to get some free publicity: I'd rather have just 50 people actually professionally looking for bugs, than 50,000 whining about everything else.
Having 500 people who genuinely test for bugs, is _worthless_ if their signal is drowned in the noise from 50,000 people posting like there's no tomorrow about how your game sucks ass because his Priest doest't _start_ with the Mages' level 50 spell. (That's sadly not even a joke. Something Awful once had a parody of an open letter to Sony, in which they asked for really ludicrious stuff, including _literally_ that a level 1 priest should start with the most powerful mage spell. Much to their surprise, they got a helluva bunch of emails aggreeing wholeheartedly.) Or how it sucks ass and is unbalanced because it doesn't _force_ everyone else to group with his Priest that bought everything _except_ healing/buff spells. (Add a long circular-backpatting whine about how players are idiots and don't appreciate how useful that priest is with his mace alone.) Etc.
And it goes downhill from there. The guy who discovered a bug and filed it, will start _one_ post. The guys arguing that their characters should have 100% resistance to damage and an insta-kill spell that costs no mana, will start one per day. And more often than not, spill into the other topics too. (Surely a post about how a mage spell sometimes fails with no explanation, not even a "your spell was interrupted" message, is _the_ right place to post about how either (A) you mages had it too good and it was about damn time that spell got a downgrade, or (B) about how we mages are the whipping boys of the devs, and they downgraded yet another of our spells. Doom, gloom, run for the hills, and all that.)
Welcome to the wonderful world of looking for the proverbial needle in the haystack.
A polar bear is a cartesian bear after a coordinate transform.
Then you have Blizzard, who didn't requested a NDA for World of Warcraft. This way, a lot of people would make movies, screenshots, features reviews, etc... Another marketing trick to create a lot of hype.
And you have Sigil, who allows only the people who registered on their forums before the beta 1 announcement. To me, that means that only people who really look after this game, so much that they bothered to register on the forums, will have the opportunity to apply. Sigil also indicated that they might conduct phone interviews with some applicants.
So, yes, a lot of people want to get in the betas just to have a free sneak peek - even if the game is far from being "done" (if that's even possible for a MMO). But I think what can make a difference is the way you select your beta testers.
...I have participated in many betas(and quite a few alphas). The one things that I have learned is that the developers never seem to listen to what the players tell them. One of the games myself and many other testers pointed out a HUGE flaw in the game balance, 4 months before it went live. It was not until several months after go live that they eventually fixed it.
The developers should be more willing to listen to testers feedback about the games. In my opinion testing is not just about finding bugs and load testing. Its about testing to see if the game is any good or not and trying to fix those issues before go live. Many of the Betas I have participated in I have never actually played after the game's release. Why? Because flaws in the game that I pointed out as making the game un-fun persist even after go live.
The best example is "Ultima Online" I played the Beta of the game from the day the beta was first available until they cut it off before go live. I filed many many bug reports, some for out right glitches, some for game play issues. Several of the glitches persisted into the second year of the games release. Nearly all of the game play issues persisted! Eventually the game just got completely fustrating and boring. I participated in the official and unofficial game forums, however many of the things that myself and many other players continuely pointed out never got addressed. Infact many changes that were implimented actually made things worse.
Power Corrupts,Absolute Power Corrupts Absolutely, leaving one person(group)in charge is absolutely corrupt.
but why even have such a process in the first place? If they honestly think they are going to get any sort of actual *testing* (I use the term loosely) from an over-hyped MMORPG community... they obviously failed basic MMORPG sociolog
Heh, but blizzard changed a lot pre-launch to post launch of wow. For those that play, the paladin class was totally revamped (many believe for the worse), and a lot of balancing issues were worked out (Will of the forsaken was originally always on, Mortal strike did 200% weapon damage...etc)
Yeah, that's what I was suspecting/wondering. Some games have been launched with such glaring errors, that, yes, I've always wondered if anyone actually tried finding any bugs during their much-hyped beta.
Heck, while you have a point that UO _is_ a good example there, I have an even better one: Anarchy Online. Read the review on Something Awful, and I can personally atest that yes, the game was _that_ broken after the devs claimed it was finally 110% fixed and stable. In fact, there's a whole slew of of more problems that Something Awful didn't even touch, like people swimming in the floor or suddenly finding themselves falling to their death from stratosphere, when a second earlier they were running on flat ground. As released, it was even worse.
I mean, it's not even minor bugs or balance issues. There were some issues which a beta test, no matter how superficial and unmotivated, simply couldn't have missed.
E.g., you could have to go through those opaque light swirls that their buggy open doors often looked like, and find yourself falling into a 6 ft hole that the random map generator had created. There just was no way out. You either tried contacting support to get you out, or (eventually) grudgingly commit suicide and respawn in the city.
Hasn't _any_ tester fallen into one of those holes? (Or even just ran into the sheer annoyance of open doors turning into a swirly graphical glitch that you can't actually see through.) I fid that totally improbable. Did everyone rather silently commit suicide than report the bug? Equally improbable. Or maybe it's simply that the developpers and publisher didn't give a rat's ass about those bug reports?
A polar bear is a cartesian bear after a coordinate transform.
Wow, now we are having bloggers post links to their own blogs to drive up traffic...and to make it worst the thing is hosted on blogspot. I mean seriously, at least get your own DNS and route it or buy some shared spaced.
Now onto the topic at hand, this just shows that 90% of bloggers are talking out of their ass. In this case he really misses the point of beta testing in an MMO. There are several very good reasons and they are reasons that are necessary to test. Later stages in almost all MMO beta test are load tests. They could care less at that point who reports bugs, they just want to make sure the servers don't go kaput when the game launches.
Another problem is you need a large number of people to beta test any MMO. There are literally hundreds of possibilities when a game is in the beta mode (if not thousands) for character development. Look at World of Warcraft. You have to test every race with every classes. You have to test quests at multiple levels, you need to test raids and dungeons. There is a lot more to beta in a MMO then in a single player or even regular multi-player game.
Not to mention the fact that the Beta taste gives you the chance to hook all those players and then start charging them. It also gives you the chance to get free word of mouth advertising. Open betas are a bit more of a joke on other games, but in those situations companies release "demos" to perform essentially the same task without calling it a beta. I have beta tested a regular game and an MMO. Let me say it is much bet to use a community of people in the tens or hundreds of thousands to test your game then to use your few hundred employees with an MMO. Like I said before, the sheer scale of MMOs makes fully closed Beta's an unreliable and ineffective means to test the game.
"Some days you just can't get rid of a bomb."
The Vanguard beta opened up to stage 1 for one reason - beta 0 people could not get groups to explore content. Sigil is not stupid enough to get the public to test their engine. But with the near infinite content available in MMORPGs it really takes 1000s of people to properly review it all. Be sure that if Drizzt_056 has done a quest for a week to get his sword of uber and doesn't get it he will report it. Probably multiple times, once in all caps. As others stated, it is also necessary for stress tests. Many games have thought they were finished and then realised their servers needed an overhaul after a stress test happened. Actually, a lot never fix this problem
They don't care if there is a leak, because when there is, they are getting that much more feedback on the game. Everytime you hear that such and such game got leaked, you know the developers are out browsing the big forums where everyone is talking about them
Sigh... now all of the good articles about VG beta won't get airtime. For shame.
I hereby dub the author of that article Mini_Jack_Thomson_09572 , for his ability to say nothing useful yet still get media attention.
My script don't crash! She crashes, you crashed her!
Most of the players treat it as a sneak preview at best or more commonly just free play.
... and then proceeds to squirrel them away for later. Then at go-live time the carefully-planned game balance is instantly pooched by these guys who run straight to the loophole and max out while the new players are still trying to find the bank.
A tiny fraction of people find and report bugs as they're supposed to.
A much larger minority actively looks for glitches, bugs, and exploits
I can see a marketing reason to open a game up to the public, but other than that, no. Besides, if your game has any sort of showstopper issue, you've lost that customer for life and a few more who that person tells about buggy this and laggy that long after you've fixed the bug.
Functional testing is best handled by people whose professional interest lies in finding and fixing bugs, not finding and exploiting bugs.
Stress testing can just as easily be accomplished by 10k macroed test sessions as by 10k live players. In fact, better, since the macros can hit everything, faster, and all at once if you want. Might as well be ready for when bot pharming teams hit your game, right? Better yet, if you find the script is leveling faster than a human player, you have a problem.
What you need to know is the fact that betas are infiltrated by those that want sneak peaks at the game. Definitely not by those that truly wish to test the product.
They're hardly being "infiltrated" by anybody. If you pre-order most games these days, you get guaranteed beta access which oddly coincides with when they're ramping up their stress tests. If you already subscribe to an online game the company is running, you tend to get beta access to their new stuff as well (whether or not you fork out cash). It's an easy way to generate word of mouth prior to release, and to make some people that over-value being part of a Beta test to get some snobbery points ("Yeah, I've been playing since Beta 2!").
You gotta figure that, given the people they're recruiting, the application process is a bit much. It's far easier to provide your email address and check a box saying you've beta tested before (and perhaps provide a timezone), and then get picked by a RNG. By the time public betas open, they don't care as much about different hardware configs. And I've seen people kicked from betas for making some exploits public in restricted forums. Never made sense to me, you plan on getting character wipes occasionally anyway so having someone publish the information shouldn't be too terrible. If you wanted to truly beta test the product, in my opinion you'd let people get developer-like access. Let some people xp/skill up the old fashioned way, let others generate their characters at the high-end of what's available in the game. Get every aspect of the play tested. That's also my problem with test servers a lot of companies make available for patch testing. It does no good to have a patch server where you'd have to level from scratch if all the changes impact the high-end game. But that's into other beefs of mine with MMORPGS...
"It is a miracle that curiosity survives formal education." -Albert Einstein
except that they dont charge for betas, as they wipe all your account after the betas ended.