Slashdot Mirror
When Is A MMORPG Beta Not A Beta?
Thanks to Skotos for its editorial discussing the definition and previous use of the 'Beta phase' in relation to MMO titles. The writer argues, overall, that "The problems [in Beta] arise because of the differing agendas of the parties involved in bringing the games to and through the Beta process." He then posits that, "well before the full featureset is in place, the complexity surpasses the point where internal QA processes are adequate to cope", but "on the other hand, for the purposes of getting high-quality feedback that tells you what is wrong and where to look in order to fix it, Beta sucks." So, it's suggested, the end result is that "[massively multiplayer] Betas become exercises in community management, usually long before the team is ready to make the transition from developing a game to operating one. Meanwhile, an increasingly jaded marketplace is judging the Beta against the same standards they judge games at launch, or even years past their launch." What, if anything, can be done to ameliorate or fix these problems?
Companies don't have a network with a million nodes of users playing the game. So they call it beta and get it into the market.
But at least they still call it beta. Microsoft releases windows and the real deal isn't the real deal until service pack 25 and 9000 hotfixes.
The real question most MMORPG gamers would ask is "When is a MMORPG not in beta?" Most games of this genre continue to add new features to the game as it progresses, thus keeping the playerbase. Everquest and its many expansions being a good example, not too many the countless things Sony has added between expansions. When are these games ever truly out of "beta". Though I do agree most beta tests dont accomplish as much as they would if they were for other types of software. Gamers tend to be gamers. They want to play the "next big game" before anyone else. That is why they sign up for betas.
"why don't you just slip into something more comfortable...like a coma!"
Seriously shouldn't a piece software (be in MMORPG or OSes) be out of beta once it is on shelves collecting money from users.
As the article says there is quite a lot of beta stages:
* Development Beta
* Playtesting Beta
* Scaling Beta
* Load Beta
but these should not be an excuse for using users as guinea pigs (esp. paying users).
You know it's not a beta any more when you stop finding "SYNTAX ERROR: LINE 6778 INVALID CHARACTER AFTER ;" in treasure caves.
Don't blame Durga. I voted for Centauri.
When a player successfully finds and reports a problem in the game, they could be awarded with extra stats, items, in-game money, etc. In fact, maybe real prizes like exclusive clothing, figurines, developer autographs, cash credit at an affiliate site, or just plain old cash would help out too.
The prizes probably should be rated by severity of the issue that they brought to light. As they find more problems, they would get more prizes.
It probably wouldn't hurt to give a special thanks page, possibly even a rankings page so that it could be turned into a true competition. The testers could receive a special title for when the game is officially released, too.
Gamers tend to like free stuff that nobody else can get and respect within the game that they're playing. Any one of these ideas should be able to feed both needs.
Until Slashdot fixes the funny modifier, use insightful or interesting. The poster knows your intentions.
And this is fine, especially for MMORPG games.
I have no idea! I play Shadowbane!
---
...and that's the bottom line! Because Stone Cold said so!!!
A company like Square-Enix could very easily maintain a stable of beta testers across 5-10 games, rotating in fresh blood every time to keep the pool alive as certain testers grow older and have less time to beta test. After each beta cycle non-responsive or poor quality beta testers could be thanked and excused from service - they could even maintain two pools, one for actual testing and one for winnowing out efficient beta testers.
Think of it like a job interview where the compensation is getting to play cutting edge games way ahead of the public as well as getting the developer's ear. And the company gets a small pool of very good testers (maybe 300) that they can count on to do good work and still be around for FFDQXXVI Online.
Yvan Eht Nioj!
It will be solved when you get publishers to stop making milestone dates based on fiscal reports and start paying attention to the development process.
Rushing to beta is just as bad as rushing to release, but many developers are forced to do it because they are working on a timeline that has little room for internal review and repeated small-scale testing.
Counter-example, Blizzard Entertainment and World of Warcraft: they have been testing internally for months, and the game is already stable and polished. When it hits beta, they will be able to manage it, because their "beta" will be more stable than some games that are on store shelves right now. That kind of development process is hard to come, and only a very few large developers have the clout to tell the distributor to STFU until the product is done.
...some betas are more beta then other betas.
The Roma Victor MMORPG project doesn't call it a beta. But they have started an equivalent.
When its ajar.
Games should have a preset 'Open Beta' for everyone who wants to download the game to participate in the test, because the most common crushing factor in a game at release is an overload on one factor in the network (usually a router somewhere down the line). This is often overlooked somewhat because the game developers are trying to perfect The Game (tm) and dont seriously factor in the network... which is surprising to see even after so many dismal launches of titles (AO, SB to name a couple).
IMHO, a beta truly ends when the servers are capable of staying online and providing a 'persistent world' for a week. That would put many games which are currently 'Live' back into beta.
1. Don't think of beta testers as unpaid labor. Most beta testers are in it for a greedy reason - to get an early look at the next big thing. Spend more time in Alpha, working with PAID testers to refine gameplay and eliminate major bugs - use Beta testing as a final "shakedown", but realize that it's more usability testing than anything else.
2. Keep it small. If you want to spend less time working on community problems and more time fixing code, don't open the beta process to everyone and their brother. Get people who are going to play and report bugs - if they don't, don't invite them to participate in the next beta.
3. Develop comprehensive crash reporting, if possible. Tell the user what went wrong where, and give them the opportunity to send it to you via email or web. I can't imagine that MS got much good feedback from Windows beta testing when the only error people got was "General Protection Fault".
4. Listen to feedback - nothing is more annoying than filing a bug report during beta and seeing the same bug in the shipping product. Have at least two people touch each bug report to avoid the "rubber stamp" syndrome.
Of course, the fact of the matter is that a lot of MMORPG's use Beta testing as an opportunity to "hook em' early". They should offer a preview edition to do this, not get people who are more interested in playing games that squashing bugs disappointed with the quality of their product.
Blogging Weight Loss, Distance Education, and more at verlin.com
The real problem with MMORPG's has nothing to do with developers. Actually it does, but it has more to do with the gamers. Currently, the trend in the MMORPG sector, across the board, is gamers expect very little from an MMORPG at live. They expect a buggy, incomplete, and reasonably content devoid game. Then they accept it because "that's how all MMORPG's are." There's only one solution to this, gamers need to STOP playing games that are not feature complete, or else developers, no matter how long the beta is will continue to turn out rehashes of EQ. The developers also need to step up to the plate and offer something new. Recently Planetside is the only game that comes to mind at being slightly innovative. Unfortunately it is about the farthest thing from a roleplaying game that is massively multiplayer. In addition to innovation the players need to stop asking for release dates and developers need to stop giving them.
Right now beta is treated as a sort of interim until release, it shouldn't be that way and developers shouldn't expect it. They know how many bugs there are, hell, they programmed the bugs in the first place. Not purposefully mind you, but they are still there. If a playerbase isn't delivering bug reports just delete accounts. It's free gaming, if they aren't doing what they applied for they should not be allowed to take part in beta. The poster above is pretty much right, the only way companies are going to get good beta testers is with incentive. If for example you find 10 or more bugs in game that need to be fixed before live, you get to keep your character, or at least get a head start or free copy of the game.
Unfortunately most of this is for naught, I myself am a jaded gamer and I don't think developers will ever change, and neither will the gamers. Long live mediocrity.
schild
editor, f13.net
I didn't join until after it released and I paid out the wazoo for the privilege. I'd say Sony has a great beta setup - it never ends and they profit off it.
When is a beta not a beta? Heck, it's better to ask when is a shipped product STILL a beta.
I've tried many times to get in on first round beta testing for MMOGs, and have given up. I have a lot of ideas for MMOGs, and I've had a lot of complaints about the current offerings. I can point to a long history of postings by me about these problems.
Not to toot my own horn too much, but I think that it would be worthwhile for developers to seek out people who have posted a lot about other games for their initial betas - people who are highly opinionated and willing to put in actual time coming up with ideas or writing up problem reportas...
Honestly, I think it depends on the individual players view as to when a MMORPG is no longer in beta - everyone is going to be looking for something different in a game. If the crafting system is fubar, then the MMOPRPG will still be in beta phase for the player who likes to craft; the PKer won't care at all (until he needs to buy phat lewt later in the game).
For me, I see a few items on my checklist until I feel I'm no longer paying to play a beta:
1. Billing system is operational: it stays up and working under a large load, and only charges me once for signing up.
2. I am able to log in first try: I get no time-outs or "could not find server" when I know that the server is up.
3. I can play for more than a few hours without being disconnected: and I'm being generous here. I don't mind being disconnected after 4 hours play because I decide to run into a full scale guild war with all my graphic settings set to high, but I shouldn't be disconnected when changing zones, or just running along an open area.
4. Quests/Missions etc can be finished: If I start a quest only to find that the NPC I need to talk to doesn't give me the item he/she should, then that's obviously a big red mark against the MMORPG. Q&A testers are there for a reason, dontyaknow...
5. Glaring gaps in gameplay are not fixed: I know over the lifetime of a MMORPG the gameplay (eg combat system, crafting) is going to be tweaked and adjusted to reflect what players are doing, or should not be doing, in the game - but if I'm playing in my first week of a MMORPG and players can't form effective groups because spells aren't anywhere near the level they should be, or a classes abilities are non-functional, then it's back to the beta drawing board for that MMORPG.
I could start getting specific now, about what I feel a MMORPG should be like upon release, but I'll just finish by saying I think it also comes down to what the MMORPG promised. If a MMORPG has promised housing - right up until release - and there is no housing, then it's still in beta. If the MMORPG has promised certain aspects to PvP or 'revolutionary' combat - and there these aspects do not exist - then it's beta, people! Why? Because I know that when these get implemeted on live servers these features that should have been in the MMORPG at release were, in fact, not there and they will also require some fixing/testing.
All this has led me to my current MMORPG of choice: Final Fantasy XI. It's been the first MMORPG in a long while where all the above criteria have been met; and then some. It was so refreshing to play a stable game at the (North American) launch.
"Meanwhile, an increasingly jaded marketplace is judging the Beta against the same standards they judge games at launch, or even years past their launch."
Well, if game companies actually gave a damn about what they claim are "finished products," where the released game didn't require as many patches as (if not more than) what a beta should be, we wouldn't be comparing the two as if there were no difference between the two classes (and really, are there?)
Me? Jaded? Nah!
It should be made ABSOLUTELY CLEAR to the end users that they are TESTING THE ENVIRONMENT. Don't allow accounts for every person, their brother, sister, dog, and cat just because you want to load test your system. Load tests are easy to simulate.
In addition to having very obvious ways (like a gigantic push button on the screen that says
"REPORT BUGS BY CLICKING HERE" or some such) to report bugs, inconsistencies, and other things.. make sure that people are given credit where credit is due, for their testing work.. and send out WEEKLY surveys to ALL testers.. and they MUST be filled out and returned within a certain time frame, otherwise the user is cancelled from the beta program.
These are ways to get quality testers. Even if they have no experience in software testing, you're at least going to get valuable input. Of course, people with software testing experience would probably be preferred, but also people with experience on other MM worlds would be great, too.. with the addition of several people who haven't the slightest clue (that's the best way to find bugs.. have people who don't know what they are SUPPOSED to do to get things to work right try things...)
Make varied groups of testers travel together, presuming it's a traditional RPG... so you have people with test experience, people with MMrpg experience, and n00bs alike teaming.. sure, there's time for free for alls, but make time required to test certain things, as well.
"Champagne for my real friends - and real pain for my sham friends!" http://ericblade.postalboard.com/
It occurs to me that the Slashdot paradigm (are we still allowed to use that word in general conversation?) would be a great way to handle bug reports. Require some proper format for bug reports and have an open forum webpage where people can post and respond to them. Then have 'moderators' who give positive or negative ratings to the original posts and the responses.
In time, the players who consistently write good bug reports, and those who provide useful feedback, such as alternate ways to reproduce a bug will gain 'karma' and thus their messages will float to the top. If moderators and submitters all had 'karma' ratings, I think you would quickly have a quite useful system.
I honestly believe that most MMORPG's are beyond what would be called 'beta'. Generally, they are in their Release stage. However, their complexity and evolution practically necessitates that there will be bugs. If I like a game, I want to help it work smoother, and as long as the bugs are rare enough that they do not interrupt my gameplay several times an hour, I would be more than happy to assist in their elimination.
An MMORPG released with no bugs will mark a significant date in the history of gaming. I haven't played enough to know, but, after all these years, has Nethack reached bug-free status?
I think a big problem with beta tests, and the preception of gamers, stems from these games ability to be internally patched. Because of autopatching, the devs feel they can release a beta way too early and then add in components as the beta progresses. That doesn't sound anything like the beta state in not-mmo games. That sounds more like an alpha state to me. Beta testing in offline games is usually about finding obscure bugs the devs would never have to time to pick up on their own and play-balancing. Betas in mmo's are about server stress-testing an unfinished product, with many game features not yet activated or horribly broken. So gamers, in their first exposure to the new game, get a broken experience and terrible lag. Beta should be about gixing a syntax error in the npc for the lvl 2 quest, or fining out the quest fails if i click on the npc 11,282 times in a row; they should not be about "this early quest-specific npc is not yet activated." These games should be in a much more finished and polished state before they go beta. Instead they get sent to cd mastering requiring massive patches just to be playable on opening day. Is it corporate's fault, rathe than the devs, jor pushin out the product too quickly. probably, but its the devs who gamers deal with in game, so as long as they prtend this isnt a problem it will continue to be so.
Beta should start when the game is usable enough to get customer input. This means that alot of content can be missing, the game can crash, there can be lots of nagging problems, but there is enough there for people to get the feel and respond.
Most people who play games, especially MMORPG are not technically savvy, you want to get input from many different people on things like UI and general mechanics of playing because later on it is more difficult to fix those problems. Once those fundamentals are done then beta can expand to address things like content, balancing, etc.
Beta should end, when the game is stable (at least stay up 1 week straight) and there is sufficient content for the majority of players. Unfortunately we live in a world where money is a driving factor, MMO games are complex and expensive so delays are very detrimental. SWG is one example of a game launched early (hmmm just before Q2 ended coincidence?) Even though the majority of the beta community protested. Not to mention the pain of AO
D6 63 0D 70 89 81 BB 8E 7B 7C 5F 5D 54 EA AB 73
Sometimes, if the beta was a good experience, I will purchase the game, as was the case with EQ and DAoC.
Betas have become not something necessarily about testing, but rather giving a preview to all of the fans who have been eagerly watching the game progress from its conception.
If a game company wants to capitalize on it, treat it as a promotional event. I'm in marketing/advertising, and from my PoV, thats exactly what these things are. For example, some games have given everybody tons of cash, lots of XP and skill points on the last day and let them go nuts. This is the kind of thing that can hook you customers. You see, if they get a taste for the high level stuff on the last day, they might be more willing to shell out the $60 for the box, and all the monthly fees necessary to get to the high level again.
Perhaps the concept of the beta-test has evolved.
Buy Steampunk Clothing Online!
Shhh, the more bugs the users find, the more bugs the developers have to fix :D
We don't want more work, now do we?
lol
Ignorance is bliss, as they say.
- shazow
Ragnarock Online its got to be in beta for atleast a 3 or 4 years by now. dont know if it ever got out of beta thu stopped playing a long time ago. then in other massily long beta's look at Mame been over 3years and still not at 1.0 yet.
Why not start a business specialiced in MMORPG Beta testing? I'm sure there is some kind of way this can work.
Example strategy:
Some helpdesk/support company can provide such a service. When they have a 'BETA Test project' they temporarly hire a couple of more agents to reduce the average load. Then they have all the agents play the game while they are not receiving phone call's and train them to correctly submit bug reports.
Just a thought, but there must be a way to make profit out of this!
I think massively multiplayer online games should offer a trial basis - 1 free or inexpensive month. Paying $70 for a CD and $15 for the first month is a hefty investment for a buggy piece of crap.
MMOG is a fledgeling genre that is dominated by relatively few products right now. Little competition equals little effort. It's a pretty safe bet any MMOG will be quite imperfect in the first 6 months of its operation. 2004 will introduce dozens and dozens of new games, each contesting for the same pool of potential subscribers. The strong will prevail, the weak will disappear.
It's been happening for the last 10 years or so, but the words 'beta test' lost all meaning, judging from the replies here as well. There used to be a decent distinction between alpha test and beta test. Alpha means it isn't yet 100% complete, beta means it's basically finished and only needs to be tested before final release, with the possibility of bug tests. A beta test copy of software used to mean that it was the release candidate. Originally a beta test didn't mean the owner knows that it's full of bugs, or that parts still have to be completed. I am surprized to see some companies do betas while they are still developing it, or have to still add the magic and only combat works. Of course users will see a beta test as a chance to play a game for free, and to play it as soon as possible. They know that out of 100 users only 5 will report bugs or even read the forum.