Slashdot Mirror

← Back to Stories (view on slashdot.org)

Exploring the Current State of Beta Testing

Posted by ScuttleMonkey on from the only-if-you-actually-learn-from-the-past dept.

Karen Hertzberg writes "Since the earliest days of MMO gaming, beta testing has played a pivotal role in the success or failure of our persistent worlds. We've come a long way since the initial tests of Ultima Online and The Realm, but what role do our current beta tests play in the potential outcomes of unreleased titles? To answer this question, Ten Ton Hammer turned to current and former beta decision makers at Cryptic Studios, NetDevil, Sony Online Entertainment, Funcom, and Mythic Entertainment. Some of their answers — and the information they reveal — may surprise you."

7 of 82 comments (clear)

  1. There Really Are Some Gems in This Article by eldavojohn · · Score: 5, Insightful
    That I wish the game companies and everyone would learn from:

    [Beta] seems to be more about marketing the game and hyping it up.

    So completely accurate.

    Honestly I feel that nowadays the developers get confused and can not find a happy medium between listening to every single testers opinion and listening to nobody. In some beta tests I have felt the community was highly ignored while in other tests I have felt that the developers tried to cater to every single tester. Both are and were recipes for failure.

    Beta tests now are glorified demos for the games. It used to be so much different. I really hate what's been done to them, and the only way to be a 'real' tester now is to get into the alpha's or in some rare cases early closed betas.

    Backing up the first quote and pointing out what Betas have become.

    I wish there were more people submitting bug reports, but that's the way it goes with beta, and weâ(TM)re still finding them regardless. Besides, I need all types [of players]. I need the exploits so we can find them ... I need the jerks.

    At first I was shocked they would want the scripters and botmakers on so early but I soon understood that you want to catch these serious things as early as possible to fix them because:

    By the time beta begins, you've made decision after decision that have compounded on each other. Your assumptions' assumptions' have assumptions about what your game is. The whole product, systems, content, operations, marketing, PR, community ramp, you name it -- is built upon them. Changing core assumptions about the product itself is unlikely to be possible without significant delays, costing progressively more money per month. (Remember, the months toward the end of the dev cycle are the most expensive ones by far.)

    Ultimately, you need to believe in your product before you conduct any sort of open beta or release a demo. You can message to players all you want that the game is a "work in progress" and that many things will change before final release, but that wonâ(TM)t stop them from making judgments about the title based on their beta experiences.

    I believe that's the fundamental reason behind Blizzard's horrid schedule slippages.

    It is disappointing to say, but testing has become a bit of a joke, and I feel that the current crop of recent games are a reflection of that. So many games are being released incomplete (as far as hyped features go) and containing issues that should have been picked up and resolved during the closed beta phase at the latest, but this isn't happening.

    It's the classic cash in while you still can mentality that has seen the release of so many unstable games only to have the servers shut off or merged down within a year.

    Here's to hoping the gaming industry finds and reads this article ... we're in a bad spot right now.

    --
    My work here is dung.
    1. Re:There Really Are Some Gems in This Article by Darinbob · · Score: 3, Insightful

      [Beta] seems to be more about marketing the game and hyping it up.

      So completely accurate.

      It has annoyed me in the past when people have said they were "lucky" to be in a beta, or grumbling that only a select few were chosen. It seems that people treat being in a beta as a perk or reward, which means that they treat the beta like an opportunity to learn the tricks before everyone else, or use it like a badge of honor. That's completely the wrong attitude, as in normal software being a beta user means a lot of hard work and attention to detail. Being in a close game beta should not be about having fun.

    2. Re:There Really Are Some Gems in This Article by DerekLyons · · Score: 2, Insightful

      You're not going to get "a lot of hard work and attention to detail" from the testers of your commercial product unless they're being paid. The fun of playing the game early is a form of payment, but if you're asking them to forgo that fun in order to only do the work part then you're insane.

      I'm guessing you've never actually part of gaming beta. I all of the ones I've been in (about nine so far), finding people willing to work hard and pay attention to detail has been pretty easy.

    3. Re:There Really Are Some Gems in This Article by fractoid · · Score: 5, Insightful

      I believe that's the fundamental reason behind Blizzard's horrid schedule slippages.

      (Warning: rabid Blizzard fanboy post ahead:) In the early days, yes, Blizzard had some pretty bad schedule slippages. Unlike most companies, they correctly identified the problem to be the schedule, not the slippages. The reason that their games are still regularly played over 10 years after release is the uncompromising attitude towards quality. That's why these days they keep everything completely dark until a game's already been in development for a couple of years, and they don't commit to a release date until a couple of months before that date.

      Games are art, and artwork isn't done until it's done. You can schedule an accounting app, because it has a strict function that's determined by an arbitrary set of rules. You can't schedule 'finishing' a game because the completion criteria are incredibly fuzzy and vauge. "It has to be fun." "The storyline needs to be engaging." "The player has to feel significant." "The world needs to be realistic." If you don't give this stuff enough time to be iterated over and refined, you end up with a bodgy game no matter how good your tech is.

      --
      Rampant carbon sequestration destroyed the Dinosaurs' tropical paradise. I'm here to help repair the damage.
    4. Re:There Really Are Some Gems in This Article by Opportunist · · Score: 2, Insightful

      It has changed, though, if you ask me.

      I've been in a stretch of MMO CBs within the last decade or so. A decade ago, you had some pretty good chance to actually know a fellow tester or two. From another game, or because of the way the CB slots were distributed.

      Today, CB slots are more often than not some sort of raffle prize. You have people who join the game forum (or, IMO worse, some MMO forum) where then some raffle is held with CB slots as the prize for the lucky winners.

      What kind of "quality" do you expect to get out of that?

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
  2. One Day... by koterica · · Score: 3, Insightful

    One Day, Beta Testing will be out of Beta.

  3. Tester for the last 10 years by le_sean_moon · · Score: 2, Insightful

    I've beta tested plenty of games over the last ten years, but I didn't realize it until after I paid for them